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(57) Abstract 

Enhanced telepresence and tetesurgery systems automatically update coordinate transformations so as to retain alignment between 
movement of an input device (700) and movement of an end effector as displayed adjacent the input device. A processor maps a controller 
workspace with an end effector workspace, and effects movement of the end effector in response to movement of the input device (700). 
This allows the use of kinematically dissimilar master and slave linkages. Gripping an input member near a gimbal point and appropriate 
input member to end effector mapping points enhance the operator's control. Dexterity is enhanced by accurately tracking orientational 
and/or angles of. movement, even if linear movement distances of the end effector do not correspond to those of the input device. 



FOR THE PURPOSES OF INFORMATION ONLY 
Codes used to identify States party to the PCT on the front pages of pamphlets publishing international applications under the PCT. 



AL 


Albania 


ES 


Spain 


LS 


Lesotho 


SI 


Slovenia 


AM 


Armenia 


FI 


Finland 


LT 


Lithuania 


SK 


Slovakia 


AT 


Austria 


FR 


France 


LU 


Luxembourg 


SN 


Senegal 


AU 


Australia 


GA 


Gabon 


LV 


Latvia 


SZ 


Swaziland 


AZ 


Azerbaijan 


GB 


United Kingdom 


MC 


Monaco 


TD 


Chad 


BA 


Bosnia and Herzegovina 


GE 


Georgia 


MD 


Republic of Moldova 


TG 


Togo 


BB 


Barbados 


GH 


Ghana 


MG 


Madagascar 


TJ 


Tajikistan 


BE 


Belgium 


GN 


Guinea 


MK 


The former Yugoslav 


TM 


Turkmenistan 


BF 


Burkina Faso 


GR 


Greece 




Republic of Macedonia 


TR 


Turkey 


BG 


Bulgaria 


HU 


Hungary 


ML 


Mali 


TT 


Trinidad and Tobago 


BJ 


Benin 


IE 


Ireland 


MN 


Mongolia 


UA 


Ukraine 


BR 


Brazil 


IL 


Israel 


MR 


Mauritania 


UG 


Uganda 


BY 


Belarus 


IS 


Iceland 


MW 


Malawi 


us 


United States of America 


CA 


Canada 


IT 


Italy 


MX 


Mexico 


UZ 


Uzbekistan 


CF 


Central African Republic 


JP 


Japan 


NE 


Niger 


VN 


Viet Nam 


CC 


Congo 


KE 


Kenya 


NL 


Netherlands 


YU 


Yugoslavia 


CH 


Switzerland 


KG 


Kyrgyzstan 


NO 


.Norway 


ZW 


Zimbabwe 


CI 


Cdte d'lvoire 


KP 


Democratic People's 


NZ 


New Zealand 






CM 


Cameroon 




Republic of Korea 


PL 


Poland 






CN 


China 


KR 


Republic of Korea 


FT 


Portugal 






cu 


Cuba 


KZ 


Kazakstan 


RO 


Romania 






cz 


Czech Republic 


LC 


Saint Lucia 


RU 


Russian Federation 






DE 


Germany 


U 


Liechtenstein 


SD 


Sudan 






DK 


Denmark 


LK 


Sri Lanka 


SE 


Sweden 






EE 


Estonia 


LR 


Liberia 


SG 


Singapore 







WO 00/60521 PCTAJS00/08526 



- 1 - 



10 



CAMERA REFERENCED CONTROL IN A 
MINIM ALLY INVASIVE SURGICAL APPARATUS 



CROSS-REFERENCE TO RELATED APPLICATIONS 
This application claims the benefit of priority from co-pending U.S. 
Provisional Patent Application Serial No. 60/128,160, filed April 7, 1999, the full 
disclosure of which is incorporated herein by reference. 



BACKGROUND OF THE INVENTION 
Minimally invasive medical techniques are aimed at reducing the amount 
of extraneous tissue which is damaged during diagnostic or surgical procedures, thereby 
15 reducing patient recovery time, discomfort, and deleterious side effects. Millions of 
surgeries are performed each year in the United States. Many of these surgeries can 
potentially be performed in a minimally invasive manner. However, only a relatively 
small number of surgeries currently use these techniques due to limitations in minimally 
invasive surgical instruments and techniques and the additional surgical training required 
20 to master them. 

Advances in minimally invasive surgical technology could dramatically 
increase the number of surgeries performed in a minimally invasive manner. The average 
length of a hospital stay for a standard surgery is significantly larger than the average 
length for the equivalent surgery performed in a minimally invasive surgical manner. 

25 Thus, the complete adoption of minimally invasive techniques could save millions of 
hospital days, and consequently millions of dollars annually in hospital residency costs 
alone. Patient recovery times, patient discomfort, surgical side effects, and time away 
from work are also reduced with minimally invasive surgery. 

The most common form of minimally invasive surgery may be endoscopy. 

30 Probably the most common form of endoscopy is laparoscopy, which is minimally 
invasive inspection and surgery inside the abdominal cavity. In standard laparoscopic 
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surgery, a patient's abdomen is insufflated with gas, and cannula sleeves are passed 
through small (approximately 1/2 inch) incisions to provide entry ports for laparoscopic 
surgical instruments. 

The laparoscopic surgical instruments generally include a laparoscope for 
viewing the surgical field, and working tools defining end effectors. Typical surgical end 
effectors include clamps, graspers, scissors, staplers, or needle holders, for example. The 
working tools are similar to those used in conventional (open) surgery, except that the 
working end or end effector of each tool is separated from its handle by, e.g., an 
approximately 12-inch long, extension tube. 

To perform surgical procedures, the surgeon passes these working tools or 
instruments through the cannula sleeves to a required internal surgical site and 
manipulates them from outside the abdomen by sliding them in and out through the 
cannula sleeves, rotating them in the cannula sleeves, levering (i.e., pivoting) the 
instruments against the abdominal wall and actuating end effectors on the distal ends of 
the instruments from outside the abdomen. The instruments pivot around centers defined 
by the incisions which extend through muscles of the abdominal wall. The surgeon 
monitors the procedure by means of a television monitor which displays an image of the 
surgical site via a laparoscopic camera. The laparoscopic camera is also introduced 
through the abdominal wall and into the surgical site. Similar endoscopic techniques are 
employed in, e.g., arthroscopy, retroperitoneoscopy, pelviscopy, nephroscopy, 
cystoscopy, cisternoscopy, sinoscopy, hysteroscopy, urethroscopy and the like. 

There are many disadvantages relating to current minimally invasive 
surgical (MIS) technology. For example, existing MIS instruments deny the surgeon the 
flexibility of tool placement found in open surgery. Most current laparoscopic tools have 
rigid shafts and difficulty is experienced in approaching the surgical site through the 
small incision. Additionally, the length and construction of many surgical instruments 
reduces the surgeon's ability to feel forces exerted by tissues and organs on the end 
effector of the associated tool. The lack of dexterity and sensitivity of surgical tools is a 
major impediment to the expansion of minimally invasive surgery. 

Minimally invasive telesurgical systems for use in surgery are being 
developed to increase a surgeon's dexterity as well as to allow a surgeon to operate on a 
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patient from a remote location. Telesurgery is a general term for surgical systems where 
the surgeon uses some form of remote control, e.g., a servomechanism, or the like, to 
manipulate surgical instrument movements rather than directly holding and moving the 
instruments by hand. In such a telesurgery system, the surgeon is provided with an image 
5 of the surgical site at the remote location. While viewing typically a three-dimensional 
image of the surgical site on a suitable viewer or display, the surgeon performs the 
surgical procedures on the patient by manipulating master control devices, at the remote 
location, which control the motion of servomechanically operated instruments. 

The servomechanism used for telesurgery will often accept input from two 

10 master controllers (one for each of the surgeon's hands), and may include two robotic 
arms. Operative communication between each master control and an associated arm and 
instrument assembly is achieved through a control system. The control system includes at 
least one processor which relays input commands from a master controller to an 
associated arm and instrument assembly and from the arm and instrument assembly to the 

15 associated master controller in the case of, e.g., force feedback. 

It would be advantageous if the position of the image capturing device 
could be changed during the course of a surgical procedure so as to enable the surgeon to 
view the surgical site from another position. It will be appreciated that, should the image 
capturing device position change, the orientation and position of the end effectors in the 

20 viewed image could also change. It would further be advantageous if the relationship in 
which end effector movement is mapped onto hand movement could again be established 
after such an image capturing device positional change. 

It is an object of the invention to provide a method and control system for 
a minimally invasive surgical apparatus which maps end effector movement onto hand 

25 movement. It is further an object of the invention to provide a method and control system 
for a minimally invasive surgical apparatus which permits the mapping of end effector 
movement onto hand movement to be reestablished after having been interrupted, for 
example, by an image capturing device positional change. 

It is to be appreciated that although the method and control system of the 

30 invention is described with reference to a minimally invasive surgical apparatus in this 
specification, the application of the invention is not to be limited to this application only, 
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but can be used in any type of apparatus where an input is entered at one location and a 
corresponding movement is required at a remote location and in which it is required, or 
merely beneficial, to map end effector orientational and positional movement onto hand 
movement through an associated master control device. 

SUMMARY OF THE INVENTION 
The invention provides enhanced telepresence and telesurgery systems 
which automatically update coordinate transformations so as to retain coordination 
between movement of an input device and movement of an end effector as displayed 
adjacent the input device. The invention generally maps a controller workspace (in which 
the input device moves) with an end effector workspace (in which the end effector 
moves), and effects movement of the end effector in response to the movement of the 
input device. This allows the use of kinematically dissimilar master and slave linkages 
having, for example, different degrees of freedom. Using an image capture device 
coupled to the end effector linkage allows calculation of the desired mapping coordinate 
transformations automatically. Input member pivot to end effector jaw pivot mapping 
enhances the operator's control, while the use of intermediate transformations allows 
portions of the kinematic train to be removed and replaced. Dexterity is enhanced by 
accurately tracking orientation and/or angles of movement, even if linear movement 
distances of the end effector do not correspond to those of the input device. 

In a first aspect, the invention provides a surgical robotic system 
comprising a master controller having an input device moveable in a controller 
workspace. A slave has a surgical end effector and actuator, the actuator moving the end 
effector in a surgical workspace in response to slave actuator signals. An imaging system 
includes an image capture device with a field of view moveable in the surgical 
workspace. The imaging system generates state variable signals indicating the field of 
view. A processor couples the master controller to the slave arm. The processor 
generates the slave actuator signals by mapping the input device in the controller 
workspace with the end effector in the surgical workspace according to a transformation. 
The processor derives the transformation in response to the state variable signals of the 
imaging system. 
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The processor will generally derive the transformation so that an image of 
the end effector in the display appears substantially connected to the input device in the 
controller workspace. The processor can determine a position and orientation of the input 
device in the master controller space from state variable signals generated by the master 
5 controller. Similarly, the processor will often determine a position and orientation of the 
end effector in the surgical workspace from the state variable signals of the slave. The 
processor can then generate the slave actuator signals by comparing the position and 
orientation of the input device and the end effector in the mapped space. 
Advantageously, this end-to-end mapping allows the use of very different kinematic 

10 trains for the master and slave systems, greatly facilitating specialized linkages such as 
those used in minimally invasive surgery. 

In many embodiments, the slave and imaging system will be coupled to 
facilitate derivation of the transformation by the processor from state variable signals 
provided from these two structures. For example, the imaging system may comprise a 

15 linkage moveably supporting the image capture device, and the slave may also comprise a 
linkage moveably supporting the end effector. The linkages may comprise joints having 
joint configurations indicated by the state variable signals. The linkages may be coupled 
in a variety of ways to facilitate derivation of the transformation by the processor. The 
coupling of the slave and imaging systems may be mechanical, electromagnetic (such as 

20 infrared), or the like. In the exemplary embodiment, the slave and imaging system 
linkages are mounted to a common base. This base may comprise a wheeled cart for 
transportation, a ceiling or wall mounted structure, an operating table, or the like. The 
state variable signals from the imaging system and/or slave need not necessarily comprise 
joint configuration or position signals, as the transformation may instead be derived from 

25 magnetic sensors (including those which can direct both location and orientation), image 
recognition-derived information, or the like. Regardless, the processor will preferably 
derive the transformation in real time, thereby allowing enhanced dexterity during and 
after image capture device movement, changes of association between masters and slaves, 
tool changes, repositioning of either the master or slave relative to the other, or the like. 

30 In another aspect, the invention provides a surgical robotic system 

comprising a master controller having an input device moveable in a controller 
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workspace. A slave has a surgical end effector and at least one actuator coupled to the 
end effector. The actuator moves the end effector in a surgical workspace in response to 
slave actuator signals. An imaging system includes an image capture device with a field 
of view moveable in the surgical workspace. The imaging system transmits an image to a 
5 display. A processor couples the master controller to the slave arm. The processor 
generates the slave actuator signals by mapping the input device in the controller 
workspace with the end effector in the surgical workspace according to a transformation. 
The processor derives the transformation so that an image of the end effector in the 
display appears substantially connected to the input device in the workspace. 

1 0 Often times, the master controller will comprise a linkage supporting the 

input device, while the slave comprises a linkage supporting the end effector, with the 
master linkage and the slave linkage being kinematically dissimilar. More specifically, 
joints of the master linkage and joints of the slave linkage will have different degrees of 
freedom, and/or the joints will define different locations in the mapped space. End-to-end 

15 mapping of the input device and end effector allow the processor to accurately generate 
the desired slave actuation signals despite these kinematic dissimilarities, which can be 
quite pronounced in specialized slave mechanisms such as those used in minimally 
invasive robotic surgery. 

In the exemplary embodiment, the processor will derive the transformation 

20 indirectly using an intermediate reference frame located at a detachable connection along 
a linkage supporting the master, end effector, and/or image capture device. This indirect 
transformation calculation significantly facilitates replacement or modification of a 
portion of the subsystem. 

The substantial connection presented to the system operator between the 

25 input device and the end effector can be enhanced by directing non-visual sensory 

information to the operator corresponding with the image on the display. The non-visual 
information will preferably indicate force and/or torque applied to the slave. While the 
information may be presented in a variety of forms, including audio, thermal, smell, or 
the like, the non-visual information will preferably be in the form of loads, forces, and/or 

30 torques applied via the input device to the hand of the operator, ideally with orientations 
substantially corresponding to the orientations of the forces and torques applied to the 
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slave (according to the image of the slave shown in the display). As described above 
regarding movement, correlation between orientations and torques on the input device in 
the slave may be revised by the controller (often when the transformation is revised) 
using end-to-end mapping. Optionally, the force and torque information presented to the 
5 operator indicates contact information (for example, engagement between an end effector 
and a tissue), disturbance information (for example, where one slave arm engages another 
slave arm outside a patient body), and/or synthetic information (including limitations on 
movements or "virtual walls" calculated in a simulated domain to prevent movement of 
an end effector in a restricted direction). Hence, the force and torque information may be 

10 derived from slave motor signals, sensors (including force sensors, pressure sensors, 
acceleration sensors, velocity sensors, or the like), simulation (including computed 
constraints), and other sources. 

In another aspect, the invention provides a surgical robotic system 
comprising a master controller having an input device supported by a linkage so that the 

1 5 input device can move in a controller workspace with a first number of degrees of 

freedom. A slave has a surgical end effector and a plurality of actuators coupled thereto 
so that the end effector can move in a surgical workspace with a second number of 
degrees of freedom in response to slave actuator signals, the second number being less 
than the first number. A processor couples the master controller to the slave. The 

20 processor generates the slave actuator signals by mapping the input device in the 

controller workspace with the end effector in the surgical workspace. This allows, for 
example, the use of masters having at least one redundant degree of freedom, or the use of 
a full six degree of freedom master with a slave having a more limited motion capability. 
Such masters can give a wide range of motion to a surgeon without constraining slave 

25 design, size, complexity, and/or end effector interchangeability. 

In yet another aspect, the invention provides a surgical robotic system 
comprising a master controller having an input device moveable in a controller 
workspace. A slave comprises a slave arm and a first tool releasably mountable on the 
arm. The first tool has a first end effector which moves in a surgical workspace in 

30 response to slave actuator signals. A second tool is releasably mounted on the slave in 
place of the first tool. The second tool has a second end effector moveable in the surgical 
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workspace in response to the slave actuator signals. The second tool is kinematically 
dissimilar to the first tool. The processor couples the master controller to the slave arm. 
The processor generates the slave actuator signals by mapping the input device in the 
controller workspace with the end effector of the mounted tool in the surgical workspace. 
5 In a still further aspect, the invention provides a surgical robotic system 

with a master controller moveable in a master controller space. The input device has a 
grip sensor for squeezing with a hand of an operator. The grip sensor defines a grip pivot. 
The slave arm has an end effector supported by a linkage so that the end effector is 
moveable in an end effector workspace. The slave arm has actuators coupled to the 

10 linkage for moving the end effector in response to slave actuator signals. The end 

effector comprises jaws with a jaw pivot. An image capture device has a field of view 
within the end effector workspace and transmits an image to a display. A processor 
couples the master controller to the slave arm, the processor generating the slave actuator 
signals in response to movement of the input device so that the jaw pivot in the display 

15 appears substantially connected with the grip pivot. 

A still further aspect of the present invention provides a surgical robotic 
system comprising a master controller having an input device moveable with a plurality 
of degrees of freedom in a master controller space. The movement of the input device 
defines at least one angle selected from the group comprising a change in angular 

20 orientation and an angle of translation. A slave arm has an end effector supported by a 
linkage with a plurality of joints so that the slave is moveable in an end effector 
workspace. The slave arm has actuators coupled to the joints for moving the end effector 
in response to slave actuator signals. An image capture device transmits an image to a 
display adjacent to the master controller. A processor couples the master controller to the 

25 slave arm. The processor generates the slave actuator signals in response to the 
movement of the input device so that at least one angle selected from the group 
comprising a change in angular orientation and an angle of translation of the end effector 
is within five degrees of the at least one angle of the input device. Advantageously, such 
angular accuracy can enhance the substantial connectedness of the input device and the 

30 end effector despite significant differences in movement distances perceived by the 
system operator. 
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In a method aspect, the invention provides a surgical robotic method 
comprising moving a master input device in a controller workspace by articulating a 

» 

plurality of master joints. A surgical end effector is moved in a surgical workspace by 
articulating a plurality of slave joints in response to slave motor signals. An image of an 
5 arbitrary field of view within the surgical workspace is displayed on a display adjacent 
the master controller. The slave motor signals are automatically generated in response to 
moving the master so that an image of the end effector in the display appears substantially 
connected with the input device in the master controller space. 

In yet another system aspect, the invention provides .a surgical robotic 

10 system comprising a master controller having an input device moveable in a master 

controller space. The input device includes first and second grip members for actuating 
with first and second digits of a hand of an operator. A slave has a surgical end effector 
that moves in a surgical workspace in response to slave actuator signals. The end effector 
includes first and second end effector elements. A processor couples the master to the 

15 slave. The processor generates the slave actuator signals so that movement of the first 
grip member substantially maps movement of the first end effector element, and so that 
movement of the second end effector element substantially maps movement of the 
second end effector element. 

The grip members and end effector elements shown in the display may 

20 optionally be substantially connected at the pivotal joints between the grip members and 
end effector elements. Alternatively, the point of substantial connectedness may be 
disposed at midpoints between the tips of the grip members and end effector elements, 
particularly when using tools having relatively long end effector element lengths between 
the pivot point and the tip. 

25 In yet another aspect, the invention provides a surgical robotic system 

comprising a master controller having a surgical handle supported by a plurality of joints 
so that the handle is moveable in a master controller space. The joints define a gimbal 
point of rotation about a plurality of axes, and the handle is adjacent the gimbal point. A 
slave has a surgical end effector which moves in a surgical workspace in response to 

30 movement of the handle. This can reduce the inertia of the master system when the 

surgeon changes orientation, particularly when the handle is substantially coincident with 



WO 00/60521 PCT/US00/08526 



~'3* 



- 10 - 



the gimbal point. Often times, a processor couples the master to the slave, and generates 
slave actuator signals so that the gimbal point of the master is substantially connected to a 
last joint of the slave adjacent the end effector. 

In yet another system aspect, the invention provides a surgical robotic 
5 system comprising a master controller having a handle which moves in a master 

controller workspace. A slave supports a surgical end effector and moves the end effector 
within a surgical workspace in response to slave actuation signals. A processor couples 
the master to the slave. The processor generates the slave actuation signals so that 
movement of a mapping point along the handle of the master controller substantially 

1 0 maps movement of a mapping point along the end effector. The processor is capable of 
changing at least one of the handle mapping point and the end effector mapping point. 

In general, the actuators may comprise a variety of motors (including 
electric, hydraulic, pneumatic, and the like). In other embodiments, the actuators may 
comprise brakes, clutches, vibrating devices which apply cycling loads using inertia, or 

1 5 the like. Still other actuators may used, particularly those which provide tactile 
stimulation in the form of heat, or the like. The tools of the present invention may 
include a variety of surgical tools and/or end effectors including forceps, grips, clamps, 
scissors, electrosurgical and mechanical scalpels, and the like. Still further end effectors 
may provide irrigation, aspiration or suction, air jets, lights, and/or imaging devices. 

20 General robotic systems are also provided (analogous to those described above), and both 
general and surgical robotic methods. 

While these systems, methods, and devices are particularly advantageous 
for robotic surgery, the present invention also encompasses similar robotic systems, 
methods, and devices for telemanipulation and telepresence in other fields and for general 

25 robotic applications. 



30 



BRIEF DESCRIPTION OF THE DRAWINGS 
The invention will now be described, by way of example, and with 
reference to the accompanying diagrammatic drawings, in which: 



WO 00/60521 



- 11 - 



PCT/US00/08526 



Figure 1 A shows a three-dimensional view of an operator station of a 
telesurgical system in accordance with the invention; 

Figure IB shows a three-dimensional view of a cart or surgical station of 
the telesurgical system, the cart carrying three robotically controlled arms, the movement 
5 of the arms being remotely controllable from the operator station shown in Figure 1 A; 

Figure 2A shows a side view of a robotic arm and surgical instrument 

assembly; 

Figure 2B shows a three-dimensional view corresponding to Figure 2A; 

Figure 3 shows a three-dimensional view of a surgical instrument; 
10 Figure 4 shows a schematic kinematic diagram corresponding to the side 

view of the robotic arm shown in Figure 2A, and indicates the arm having been displaced 
from one position into another position; 

Figure 5 shows, at an enlarged scale, a wrist member and end effector of 
the surgical instrument shown in Figure 3, the wrist member and end effector being 
15 movably mounted on a working end of a shaft of the surgical instrument; 

Figure 6A shows a three-dimensional view of a hand held part or wrist 
gimbal of a master control device of the telesurgical system; 

Figure 6B shows a three-dimensional view of an articulated arm portion of 
the master control device of the telesurgical system on which the wrist gimbal of Figure 
20 6A is mounted in use; 

Figure 6C shows a three-dimensional view of the master control device 
showing the wrist gimbal of Figure 6 A mounted on the articulated arm portion of Figure 
6B; 

Figure 7 shows a schematic three-dimensional drawing indicating the 
25 positions of the end effectors relative to a viewing end of an endoscope and the 

corresponding positions of master control devices relative to the eyes of an operator, 
typically a surgeon; 

Figure 8 shows a schematic three-dimensional drawing indicating the 
position and orientation of an end effector relative to a camera Cartesian coordinate 
30 reference system; 
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Figure 9 shows a schematic three-dimensional drawing indicating the 
position and orientation of a pincher formation of the master control device relative to an 
eye Cartesian coordinate reference system; 

Figure 1 0 shows a schematic side view of part of the surgical station of the 
5 minimally invasive surgical apparatus indicating the location of Cartesian reference 

coordinate systems used by a control system of the minimally invasive surgical apparatus 
to determine the position and orientation of an end effector relative to a Cartesian 
reference coordinate system at the viewing end of an image capturing device; 

Figure 1 1 shows a schematic side view of part of the operator station of the 
1 0 minimally invasive surgical apparatus indicating the location of Cartesian reference 
coordinate systems used by the control system of the minimally invasive surgical 
apparatus to determine the position and orientation of the pincher formation of the master 
control device relative to an eye Cartesian reference coordinate system; 

Figure 1 1 A-C schematically illustrates corresponding mapping locations 
15 on the surgeon's hand, on the master controller, and on the end effector and methods for 
their selection; 

Figure 12 schematically illustrates a high level control architecture model 
of a master/slave surgical robotic system; 

Figure 12A shows a schematic block diagram indicating steps followed by 
20 the control system of the minimally invasive surgical apparatus in determining end 

effector position and orientation relative to the Cartesian reference coordinate system at 
the viewing end of the image capturing device; 

Figure 13 shows a schematic block diagram indicating steps followed by 
the control system of the minimally invasive surgical apparatus in determining pincher 
25 formation position and orientation relative to the eye Cartesian reference coordinate 
system; 

Figure 14 shows a block diagram representing control steps followed by 
the control system of the minimally invasive surgical apparatus in effecting control 
between pincher formation positional and orientational movement and end effector 
30 positional and orientational movement; 
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Figure 15 shows further detail of a "simulated domain" of the control 
system shown in Figure 14; 

Figure 16 shows one embodiment of a simulation block shown in 

Figure 15; 

5 Figure 1 7 shows a relationship between L and 1/L; 

Figure 18 shows another embodiment of the simulation block shown in 

Figure 15; 

Figure 19 shows a block diagram indicating the imposition of simulated 
velocity and position limits; 
10 Figure 20 shows a preferred embodiment of the simulation block shown in 

Figure 15; and 

Figure 21 shows a block diagram indicating the imposition of simulated 
velocity and position limits relating to orientational slave movement. 

1 5 DESCRIPTION OF THE SPECIFIC EMBODIMENTS 

This application is related to the following patents and patent applications, 
the full disclosures of which are incorporated herein by reference: PCT International 
Application No. PCT/US98/19508, entitled "Robotic Apparatus", filed on September 18, 
1998, U.S. Application Serial No. 60/1 1 1,713, entitled "Surgical Robotic Tools, Data 

20 Architecture, and Use", filed on December 8, 1998; U.S. Application Serial 

No. 60/1 1 1,71 1, entitled "Image Shifting for a Telerobotic System", filed on December 8, 
1998; U.S. Application Serial No. 60/1 1 1,714, entitled "Stereo Viewer System for Use in 
Telerobotic System", filed on December 8, 1998; U.S. Application Serial No. 60/111,710, 
entitled "Master Having Redundant Degrees of Freedom", filed on December 8, 1998, 

25 U.S. Application No. 60/1 16,891, entitled "Dynamic Association of Master and Slave in a 
Minimally Invasive Telesurgery System", filed on January 22, 1999; and U.S. Patent No. 
5,808,665, entitled "Endoscopic Surgical Instrument and Method for Use", issued on 
September 15, 1998. 

As used herein, objects (and/or images) appear "substantially connected" if 

30 a direction of an incremental positional movement of a first object matches the direction 
of an incremental positional movement of the second object (often as seen in an image). 
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Matching directions need not be exactly equal, as the objects (or the object and the image) 
may be perceived as being connected if the angular deviation between the movements 
remains less than about ten degrees, preferably being less than about five degrees. 
Similarly, objects and/or images may be perceived as being "substantially and 
5 orientationally connected" if they are substantially connected and if the direction of an 
incremental orientational movement of the first object is matched by the direction of an 
incremental orientational movement of the second object (often as seen in an image 
displayed near the first object). 

Additional levels of connectedness may, but need not, be provided. 

10 "Magnitude connection" indicates substantial connection and that the magnitude of 

orientational and/or positional movements of the first object and second object (typically 
as seen in an image) are directly related. The magnitudes need not be equal, so that it is 
possible to accommodate scaling and/or warping within a substantially magnitude 
connected robotic system. Orientational magnitude connection will imply substantial and 

15 orientational connection as well as related orientational movement magnitudes, while 
substantial and magnitude connection means substantial connection with positional 
magnitudes being related. 

As used herein, a first object appears absolutely positionally connected 
with an image of a second object if the objects are substantially connected and the 

20 position of the first object and the position of the image of the second object appear to 
match, i.e., to be at the same location, during movement. A first object appears 
absolutely orientationally connected with an image of the second object if they are 
substantially connected and the orientation of the first object and the second object 
appear to match during movement. 

25 Referring to Figure 1 A of the drawings, an operator station or surgeon's 

console of a minimally invasive telesurgical system is generally indicated by reference 
numeral 200. The station 200 includes a viewer 202 where an image of a surgical site is 
displayed in use. A support 204 is provided on which an operator, typically a surgeon, 
can rest his or her forearms while gripping two master controls (not shown in Figure 1 A), 

30 one in each hand. The master controls are positioned in a space 206 inwardly beyond the 
support 204. When using the control station 200, the surgeon typically sits in a chair in 
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front of the control station 200, positions his or her eyes in front of the viewer 202 and 
grips the master controls one in each hand while resting his or her forearms on the 
support 204. 

In Figure IB of the drawings, a cart or surgical station of the telesurgical 
5 system is generally indicated by reference numeral 300. In use, the cart 300 is positioned 
close to a patient requiring surgery and is then normally caused to remain stationary until 
a surgical procedure to be performed has been completed. The cart 300 typically has 
wheels or castors to render it mobile. The station 200 is typically positioned remote from 
the cart 300 and can be separated from the cart 300 by a great distance, even miles away, 
1 0 but will typically be used within an operating room with the cart 300. 

The cart 300 typically carries three robotic arm assemblies. One of the 

robotic arm assemblies, indicated by reference numeral 302, is arranged to hold an image 

> 

capturing device 304, e.g., an endoscope, or the like. Each of the two other arm 
assemblies 10, 10 respectively, includes a surgical instrument 14. The endoscope 304 

15 has a viewing end 306 at a remote end of an elongate shaft thereof. It will be appreciated 
that the endoscope 304 has an elongate shaft to permit its viewing end 306 to be inserted 
through an entry port into an internal surgical site of a patient's body. The endoscope 304 
is operatively connected to the viewer 202 to display an image captured at its viewing end 
306 on the viewer 202. Each robotic arm assembly 10, 10 is normally operatively 

20 connected to one of the master controls. Thus, the movement of the robotic arm 

assemblies 10, 10 is controlled by manipulation of the master controls. The instruments 
14 of the robotic arm assemblies 10, 10 have end effectors which are mounted on wrist 
members which are pivotally mounted on distal ends of elongate shafts of the instruments 
14, as is described in greater detail hereinbelow. It will be appreciated that the 

25 instruments 14 have elongate shafts to permit the end effectors to be inserted through 
entry ports into the internal surgical site of a patient's body. Movement of the end 
effectors relative to the ends of the shafts of the instruments 14 is also controlled by the 
master controls. 

The robotic arms 10, 10, 302 are mounted on a carriage 97 by means of 
30 setup joint arms 95. The carriage 97 can be adjusted selectively to vary its height relative 
to a base 99 of the cart 300, as indicated by arrows K. The setup joint arms 95 are 
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arranged to enable the lateral positions and orientations of the arms 10, 10, 302 to be 
varied relative to a vertically extending column 93 of the cart 300. Accordingly, the 
positions, orientations and heights of the arms 10, 10, 302 can be adjusted to facilitate 
passing the elongate shafts of the instruments 14 and the endoscope 304 through the entry 
5 ports to desired positions relative to the surgical site. When the surgical instruments 14 
and endoscope 304 are so positioned, the setup joint arms 95 and carriage 97 are typically 
locked in position. 

In Figures 2A and 2B of the drawings, one of the robotic arm assemblies 
10 is shown in greater detail. Each assembly 10 includes an articulated robotic arm 12, 
10 and a surgical instrument, schematically and generally indicated by reference numeral 14, 
mounted thereon. Figure 3 indicates the general appearance of the surgical instrument 14 
in greater detail. 

The surgical instrument 14 includes an elongate shaft 14.1. The wrist-like 
mechanism, generally indicated by reference numeral 50, is located at a working end of 

15 the shaft 14.1. A housing 53, arranged releasably to couple the instrument 14 to the 
robotic arm 12, is located at an opposed end of the shaft 14.1. In Figure 2 A, and when 
the instrument 14 is coupled or mounted on the robotic arm 12, the shaft 14.1 extends 
along an axis indicated at 14.2. The instrument 14 is typically releasably mounted on a 
carriage 1 1 , which can be driven to translate along a linear guide formation 24 of the aim 

20 12 in the direction of arrows P. 

The robotic arm 12 is typically mounted on a base or platform at an end of 
its associated setup joint arm 95 by means of a bracket or mounting plate 16. 

The robotic arm 12 includes a cradle, generally indicated at 18, an upper 
arm portion 20, a forearm portion 22 and the guide formation 24. The cradle 18 is 

25 pivotally mounted on the plate 16 in a gimbaled fashion to permit rocking movement of 
the cradle 18 in the direction of arrows 26 as shown in Figure 2B, about a pivot axis 28. 
The upper arm portion 20 includes link members 30, 32 and the forearm portion 22 
includes link members 34, 36. The link members 30, 32 are pivotally mounted on the 
cradle 18 and are pivotally connected to the link members 34, 36. The link members 34, 

30 36 are pivotally connected to the guide formation 24. The pivotal connections between 
the link members 30, 32, 34, 36, the cradle 18, and the guide formation 24 are arranged to 
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constrain the robotic arm 12 to move in a specific manner. The movement of the robotic 
arm 12 is illustrated schematically in Figure 4. 

With reference to Figure 4, the solid lines schematically indicate one 
position of the robotic arm and the dashed lines indicate another possible position into 
5 which the arm can be displaced from the position indicated in solid lines. 

It will be understood that the axis 14.2 along which the shaft 14.1 of the 
instrument 14 extends when mounted on the robotic arm 12 pivots about a pivot center or 
fulcrum 49. Thus, irrespective of the movement of the robotic arm 12, the pivot center 49 
normally remains in the same position relative to the stationary cart 300 on which the arm 

10 12 is mounted. In use, the pivot center 49 is positioned at a port of entry into a patient's 
body when an internal surgical procedure is to be performed. It will be appreciated that 
the shaft 14.1 extends through such a port of entry, the wrist-like mechanism 50 then 
being positioned inside the patient's body. Thus, the general position of the 
mechanism 50 relative to the surgical site in a patient's body can be changed by 

15 movement of the arm 12. Since the pivot center 49 is coincident with the port of entry, 
such movement of the arm does not excessively effect the surrounding tissue at the port 
of entry. 

As cian best be seen with reference to Figure 4, the robotic arm 12 provides 
three degrees of freedom of movement to the surgical instrument 14 when mounted 

20 thereon. These degrees of freedom of movement are firstly the gimbaled motion 

indicated by arrows 26, pivoting or pitching movement as indicated by arrows 27 and the 
linear displacement in the direction of arrows P. Movement of the arm as indicated by 
arrows 26, 27 and P is controlled by appropriately positioned actuators, e.g., electrical 
motors, or the like, which respond to inputs from its associated master control to drive the 

25 arm 12 to a desired position as dictated by movement of the master control. 

Appropriately positioned sensors, e.g., potentiometers, encoders, or the like, are provided 
on the arm and its associated setup joint arm 95 to enable a control system of the 
minimally invasive telesurgical system to determine joint positions, as described in 
greater detail hereinbelow. It will be appreciated that whenever "sensors" are referred to 

30 in this specification, the term is to be interpreted widely to include any appropriate 

sensors such as positional sensors, velocity sensors, or the like. It will be appreciated that 
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by causing the robotic aim 12 selectively to displace from one position to another, the 
general position of the wrist-like mechanism 50 at the surgical site can be varied during 
the performance of a surgical procedure. 

Referring now to Figure 5 of the drawings, the wrist-like mechanism 50 
5 will now be described in greater detail. In Figure 5, the working end of the shaft 14.1 is 
indicated at 14.3. The wrist-like mechanism 50 includes a wrist member 52. One end 
portion of the wrist member 52 is pivotally mounted in a clevis, generally indicated at 17, 
on the end 14.3 of the shaft 14.1 by means of a pivotal connection 54. The wrist member 
52 can pivot in the direction of arrows 56 about the pivotal connection 54. An end 

10 effector, generally indicated by reference numeral 58, is pivotally mounted on an opposed 
end of the wrist member 52. The end effector 58 is in the form of, e.g., a clip applier for 
anchoring clips during a surgical procedure. Accordingly, the end effector 58 has two 
parts 58.1, 58.2 together defining a jaw-like arrangement. 

It will be appreciated that the end effector can be in the form of any 

15 desired surgical tool, e.g., having two members or fingers which pivot relative to each 
other, such as scissors, pliers for use as needle drivers, or the like. Instead, it can include 
a single working member, e.g., a scalpel, cautery electrode, or the like. When a tool other 
than a clip applier is desired during the surgical procedure, the tool 14 is simply removed 
from its associated arm and replaced with an instrument bearing the desired end effector, 

20 e.g., a scissors, or pliers, or the like. 

The end effector 58 is pivotally mounted in a clevis, generally indicated by 
reference numeral 19, on an opposed end of the wrist member 52, by means of a pivotal 
connection 60. It will be appreciated that free ends 1 1, 13 of the parts 58.1, 58.2 are 
angularly displaceable about the pivotal connection 60 toward and away from each other 

25 as indicated by arrows 62, 63. It will further be appreciated that the members 58. 1 , 58.2 
can be displaced angularly about the pivotal connection 60 to change the orientation of 
the end effector 58 as a whole, relative to the wrist member 52. Thus, each part 58.1, 
58.2 is angularly displaceable about the pivotal connection 60 independently of the other, 
so that the end effector 58, as a whole, is angularly displaceable about the pivotal 

30 connection 60 as indicated in dashed lines in Figure 5. Furthermore, the shaft 14.1 is 

rotatably mounted on the housing 53 for rotation as indicated by the arrows 59. Thus, the 
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end effector 58 has three degrees of freedom of movement relative to the arm 12, namely, 
rotation about the axis 14.2 as indicated by arrows 59, angular displacement as a whole 
about the pivot 60 and angular displacement about the pivot 54 as indicated by arrows 56. 
By moving the end effector within its three degrees of freedom of movement, its 
5 orientation relative to the end 14.3 of the shaft 14.1 can selectively be varied. It will be 
appreciated that movement of the end effector relative to the end 14.3 of the shaft 14.1 is 
controlled by appropriately positioned actuators, e.g., electrical motors, or the like, which 
respond to inputs from the associated master control to drive the end effector 58 to a 
desired orientation as dictated by movement of the master control. Furthermore, 

10 appropriately positioned sensors, e.g., encoders, or potentiometers, or the like, are 
provided to permit the control system of the minimally invasive telesurgical system to 
determine joint positions as described in greater detail hereinbelow. 

One of the master controls 700, 700 is indicated in Figure 6C of the 
drawings. A hand held part or wrist gimbal of the master control device 700 is indicated 

15 in Figure 6A and is generally indicated by reference numeral 699. Part 699 has an 

articulated arm portion including a plurality of members or links 702 connected together 
by pivotal connections or joints 704. The surgeon grips the part 699 by positioning his or 
her thumb and index finger over a pincher formation 706. The surgeon's thumb and index 
finger are typically held on the pincher formation 706 by straps (not shown) threaded 

20 through slots 710. When the pincher formation 706 is squeezed between the thumb and 
index finger, the fingers or end effector elements of the end effector 58 close. When the 
thumb and index finger are moved apart the fingers of the end effector 58 move apart in 
sympathy with the moving apart of the pincher formation 706. The joints of the part 699 
are operatively connected to actuators, e.g., electric motors, or the like, to provide for, 

25 e.g., force feedback, gravity compensation, and/or the like, as described in greater detail 
hereinbelow. Furthermore, appropriately positioned sensors, e.g., encoders, or 
potentiometers, or the like, are positioned on each joint 704 of the part 699, so as to 
enable joint positions of the part 699 to be determined by the control system. 

The part 699 is typically mounted on an articulated arm 712 as indicated in 

30 Figure 6B. Reference numeral 4 in Figures 6A and 6B indicates the positions at which 
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the part 699 and the articulated arm 712 are connected together. When connected 
together, the part 699 can displace angularly about an axis at 4. 

The articulated arm 712 includes a plurality of links 714 connected 
together at pivotal connections or joints 716. It will be appreciated that also the 
5 articulated arm 712 has appropriately positioned actuators, e.g., electric motors, or the 
like, to provide for, e.g., force feedback, gravity compensation, and/or the like. 
Furthermore, appropriately positioned sensors, e.g., encoders, or potentiometers, or the 
like, are positioned on the joints 716 so as to enable joint positions of the articulated arm 
712 to be determined by the control system as described in greater detail hereinbelow. 

1 0 To move the orientation of the end effector 58 and/or its position along a 

translational path, the surgeon simply moves the pincher formation 706 to cause the end 
effector 58 to move to where he wants the end effector 58 to be in the image viewed in 
the viewer 202. Thus, the end effector position and/or orientation is caused to follow that 
of the pincher formation 706. 

15 The master control devices 700, 700 are typically mounted on the station 

200 through pivotal connections at 717 as indicated in Figure 6B. As mentioned 
hereinbefore, to manipulate each master control device 700, the surgeon positions his or 
her thumb and index finger over the pincher formation 706. The pincher formation 706 is 
positioned at a free end of the part 699 which in turn is mounted on a free end of the 

20 articulated arm portion 712. 

The electric motors and sensors associated with the robotic arms 12 and 
the surgical instruments 14 mounted thereon, and the electric motors and sensors 
associated with the master control devices 700 are operatively linked in the control 
system. The control system typically includes at least one processor, typically a plurality 

25 of processors, for effecting control between master control device input and responsive 
robotic arm and surgical instrument output and for effecting control between robotic arm 
and surgical instrument input and responsive master control output in the case of, e.g., 
force feedback. 

In use, and as schematically indicated in Figure 7 of the drawings, the 
30 surgeon views the surgical site through the viewer 202. The end effector 58 carried on 
each arm 12 is caused to perform positional and orientational movements in response to 
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movement and action inputs on its associated master controls. The master controls are 
indicated schematically at 700, 700. It will be appreciated that during a surgical 
procedure images of the end effectors 58 are captured by the endoscope 304 together with 
the surgical site and are displayed on the viewer 202 so that the surgeon sees the 
5 responsive movements and actions of the end effectors 58 as he or she controls such 
movements and actions by means of the master control devices 700, 700. The control 
system is arranged to cause end effector orientational and positional movement as viewed 
in the image at the viewer 202 to be mapped onto orientational and positional movement 
of a pincher formation of the master control as will be described in . greater detail 
10 hereinbelow. 

The operation of the control system of the minimally invasive surgical 
apparatus will now be described in greater detail. In the description which follows, the 
control system will be described with reference to a single master control 700 and its 
associated robotic arm 12 and surgical instrument 14. The master control 700 will be 

15 referred to simply as "master" and its associated robotic arm 12 and surgical instrument 
14 will be referred to simply as "slave." 

The method whereby control between master movement and 
corresponding slave movement is achieved by the control system of the minimally 
invasive surgical apparatus will now be described with reference to Figures 7 to 9 of the 

20 drawings in overview fashion. The method will then be described in greater detail with 
reference to Figures 10 to 21 of the drawings. 

Control between master and slave movement is achieved by comparing 
master position and orientation in an eye Cartesian coordinate reference system with slave 
position and orientation in a camera Cartesian coordinate reference system. For ease of 

25 understanding and economy of words, the term "Cartesian coordinate reference system" 
will simply be referred to as "frame" in the rest of this specification. Accordingly, when 
the master is stationary, the slave position and orientation within the camera frame is 
compared with the master position and orientation in the eye frame, and should the 
position and/or orientation of the slave in the camera frame not correspond with the 

30 position and/or orientation of the master in the eye frame, the slave is caused to move to a 
position and/or orientation in the camera frame at which its position and/or orientation in 
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the camera frame does correspond with the position and/or orientation of the master in the 
eye frame. In Figure 8, the camera frame is generally indicated by reference numeral 610 
and the eye frame is generally indicated by reference numeral 612 in Figure 9. 

When the master is moved into a new position and/or orientation in the eye 
5 frame 612, the new master position and/or orientation does not correspond with the 

previously corresponding slave position and/or orientation in the camera frame 610. The 
control system then causes the slave to move into a new position and/or orientation in the 
camera frame 610 at which new position and/or orientation, its position and orientation in 
the camera frame 610 does correspond with the new position and/or orientation of the 

1 0 master in the eye frame 612. 

It will be appreciated that the control system includes at least one, and 
typically a plurality, of processors which compute new corresponding positions and 
orientations of the slave in response to master movement input commands on a continual 
basis determined by the processing cycle rate of the control system. A typical processing 

15 cycle rate of the control system under discussion is about 1300 Hz. Thus, when the 
master is moved from one position to a next position, the corresponding movement 
desired by the slave to respond is computed at about 1300 Hz. Naturally, the control 
system can have any appropriate processing cycle rate depending on the processor or 
processors used in the control system. All real-time servocycle processing is preferably 

20 conducted on a DSP (Digital Signal Processor) chip. DSPs are preferable because of their 
constant calculation predictability and reproducibility. A Share DSP from Analog 
Devices, Inc. of Massachusetts is an acceptable example of such a processor for 
performing the functions described herein. 

The camera frame 610 is positioned such that its origin 614 is positioned at 

25 the viewing end 306 of the endoscope 304. Conveniently, the z axis of the camera frame 
610 extends axially along a viewing axis 616 of the endoscope 304. Although in Figure 8, 
the viewing axis 616 is shown in coaxial alignment with a shaft axis of the endoscope 
304, it is to be appreciated that the viewing axis 616 can be angled relative thereto. Thus, 
the endoscope can be in the form of an angled scope. Naturally, the x and y axes are 

30 positioned in a plane perpendicular to the z axis. The endoscope is typically angularly 
displaceable about its shaft axis. The x, y and z axes are fixed relative to the viewing axis 
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of the endoscope 304 so as to displace angularly about the shaft axis in sympathy with 
angular displacement of the endoscope 304 about its shaft axis. 

To enable the control system to determine slave position and orientation, a 
frame is defined on or attached to the end effector 58. This frame is referred to as an end 
5 effector frame or slave tip frame, in the rest of this specification, and is generally 
indicated by reference numeral 618. The end effector frame 61 8 has its origin at the 
pivotal connection 60. Conveniently, one of the axes e.g. the z axis, of the frame 61 8 is 
defined to extend along an axis of symmetry, or the like, of the end effector 58. 
Naturally, the x and y axes then extend perpendicularly to the z axis. It will appreciated 

10 that the orientation of the slave is then defined by the orientation of the frame 618 having 
its origin at the pivotal connection 60, relative to the camera frame 610. Similarly, the 
position of the slave is then defined by the position of the origin of the frame at 60 
relative to the camera frame 610. 

Referring now to Figure 9 of the drawings, the eye frame 612 is chosen 

15 such that its origin corresponds with a position 201 where the surgeon's eyes are normally 
located when he or she is viewing the surgical site at the viewer 202. The z axis extends 
along a line of sight of the surgeon, indicated by axis 620, when viewing the surgical site 
through the viewer 202. Naturally, the x and y axes extend perpendicularly from the z 
axis at the origin 201. Conveniently, the y axis is chosen to extend generally vertically 

20 relative to the viewer 202 and the x axis is chosen to extend generally horizontally 
relative to the viewer 202. 

To enable the control system to determine master position and orientation 
within the viewer frame 612, a point on the master is chosen which defines an origin of a 
master or master tip frame, indicated by reference numeral 622. This point is chosen at a 

25 point of intersection indicated by reference numeral 3 A between axes of rotation 1 and 3 
of the master, as can best be seen in Figure 6A of the drawings. Conveniently, the z axis 
of the master frame 622 on the master extends along an axis of symmetry of the pincher 
formation 706 which extends coaxially along the rotational axis 1 . The x and y axes then 
extend perpendicularly from the axis of symmetry 1 at the origin 3A. Accordingly, 

30 orientation of the master within the eye frame 612 is defined by the orientation of the 
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master frame 622 relative to the eye frame 612. The position of the master in the eye 
frame 612 is defined by the position of the origin 3 A relative to the eye frame 612. 

How the position and orientation of the slave within the camera frame 610 
is determined by the control system will now be described with reference to Figure 10 of 
5 the drawings. Figure 10 shows a schematic diagram of one of the robotic arm 12 and 
surgical instrument 14 assemblies mounted on the cart 300. However, before 
commencing with a description of Figure 10, it is appropriate to describe certain 
previously mentioned aspects of the surgical station 300 which impact on the 
determination of the orientation and position of the slave relative to the camera frame 
10 610. 

In use, when it is desired to perform a surgical procedure by means of the 
minimally invasive surgical apparatus, the surgical station 300 is moved into close 
proximity to a patient requiring the surgical procedure. The patient is normally supported 
on a surface such as an operating table, or the like. To make allowance for support 

15 surfaces of varying height, and to make allowance for different positions of the surgical 
station 300 relative to the surgical site at which the surgical procedure is to be performed, 
(he surgical station 300 is provided with the ability to have varying initial setup 
configurations. Accordingly, the robotic arms 12, 12, and the endoscope arm 302 are 
mounted on the carriage 97 which is heightwise adjustable, as indicated by arrows K, 

20 relative to the base 99 of the cart 300, as can best be seen in Figures IB and 1 0 of the 
drawings. Furthermore, the robotic arms 12, 12 and the endoscope arm 302 are mounted 
on the carriage 97 by means of the setup joint arms 95. Thus, the lateral position and 
orientation of the arms 12, 12, 302 can be selected by moving the setup joint arms 95. 
Thus, at the commencement of the surgical procedure, the cart 300 is moved into the 

25 position in close proximity to the patient, an appropriate height of the carriage 97 is 
selected by moving it to an appropriate height relative to the base 99 and the surgical 
instruments 14 are moved relative to the carriage 97 so as to introduce the shafts of the 
instruments 14 and the endoscope 304 through the ports of entry and into positions in 
which the end effectors 58 and the viewing end 306 of the endoscope 304 are 

30 appropriately positioned at the surgical site and the fulcrums are coincident with the ports 
of entry. Once the height and positions are selected, the carriage 97 is locked at its 
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appropriate height and the setup joint anns 95 are locked in their positions and 
orientations. Normally, throughout the surgical procedure, the carriage 97 is maintained 
at the selected height and similarly the setup joint arms 95 are maintained in their selected 
positions. However, if desired, either the endoscope or one or both of the instruments can 
5 be introduced through other ports of entry during the surgical procedure. 

Returning now to Figure 10, the determination by the control system of the 
position and orientation of the slave within the camera frame 610 will now be described. 
It will be appreciated that this is achieved by means of one or more processors having a 
specific processing cycle rate. Thus, where appropriate, whenever position and 

10 orientation are referred to in this specification, it should be borne in mind that a 

corresponding velocity is also readily determined. The control system determines the 
position and orientation of the slave within the camera frame 610 by determining the 
position and orientation of the slave relative to a cart frame 624 and by determining the 
orientation and position of the endoscope 304 with reference to the same cart frame 624. 

15 The cart frame 624 has an origin indicated by reference numeral 626 in Figure 10. 

To determine the position and orientation of the slave relative to the cart 
frame 624, the position of a fulcrum frame 630 having its origin at the fulcrum 49 is 
determined within the cart frame 624 as indicated by the arrow 628 in dashed lines. It 
will be appreciated that the position of the fulcrum 49 normally remains at the same 

20 location, coincident with a port of entry into the surgical site, throughout the surgical 
procedure. The position of the end effector frame 61 8 on the slave, having its origin at 
the pivotal connection 60, is then determined relative to the fulcrum frame 630 and the 
orientation of the end effector frame 618 on the slave is also determined relative to the 
fulcrum frame 630. The position and orientation of the end effector frame 61 8 relative to 

25 the cart frame is then determined by means of routine calculation using trigonometric 
relationships. 

It will be appreciated that the robotic arm 302 of the endoscope 304 is 
constrained to move in similar fashion to the robotic arm 10, as indicated schematically in 
Figure 4 of the drawings. Thus, the endoscope 304 when positioned with its viewing end 
30 306 directed at the surgical site, also defines a fulcrum coincident with its associated port 
of entry into the surgical site. The endoscope arm 302 can be driven to cause the 
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endoscope 304 to move into a different position during a surgical procedure, to enable the 
surgeon to view the surgical site from a different position in the course of performing the 
surgical procedure. It will be appreciated that movement of the viewing end 306 of the 
endoscope 304 is performed by varying the orientation of the endoscope 304 relative to 
5 its pivot center or fulcrum. The position and orientation of the camera frame 610 within 
the cart frame 624 is determined in similar fashion to the position and orientation of the 
slave within the cart frame 624. When the position and orientation of the camera frame 
610 relative to the cart frame 624, and the position and orientation of the slave relative to 
the cart frame 624 have been determined in this manner, the position and the orientation 

10 of the slave relative to the camera frame 61 0 is readily determinable through routine 
calculation using trigonometric relationships. 

How the position and orientation of the master within the viewer frame 
612 is determined by the control system will now be described with reference to Figure 
1 1 of the drawings. Figure 1 1 shows a schematic diagram of one of the master controls 

1 5 700 at the operator station 200. 

The operator station 200 optionally also includes setup joint arms, as 
indicated at 632, to enable the general location of the masters 700, 700 to be varied to suit 
the surgeon. Thus, the general position of the masters 700, 700 can be selectively varied 
to bring the masters 700, 700 into a general position at which they are comfortably 

20 positioned for the surgeon. When the masters 700, 700 are thus comfortably positioned, 
the setup joint arms 632 are locked in position and are normally maintained in that 
position throughout the surgical procedure. 

To determine the position and orientation of the master 700, as indicated in 
Figure 11, within the eye frame 612, the position and orientation of the eye frame 612 

25 relative to a surgeon's station frame 634, and the position and orientation of the master 
700 relative to the surgeon's frame 634 is determined. The surgeon's station frame 634 
has its origin at a location which is normally stationary during the surgical procedure, and 
is indicated at 636. 

To determine the position and orientation of the master 700 relative to the 

30 station frame 634, a position of a master setup frame 640 at an end of the setup joint arms 
632 on which the master 700 is mounted, relative to the station frame 636, is determined, 
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as indicated by the arrow 638 in dashed lines. The position and orientation of the master 
frame 622 on the master 700 having its origin at 3A is then determined relative to the 
master setup frame 640. In this manner, the position and orientation of the master frame 
622 relative to the frame 634 can be determined by means of routine calculation using 
5 trigonometric relationships. The position and orientation of the eye frame 612 relative to 
the station frame 634 is determined in similar fashion. It will be appreciated that the 
position of the viewer 202 relative to the rest of the surgeon's console 200 can selectively 
be varied to suit the surgeon. The position and orientation of the master frame 622 
relative to the eye frame 612 can then be determined from the position and orientation of 

10 the master frame 622 and the eye frame 612 relative to the surgeon station frame 634 by 
means of routine calculation using trigonometric relationships. 

In the manner described above, the control system of the minimally 
invasive surgical apparatus determines the position and orientation of the end effector 58 
by means of the end effector frame 618 in the camera frame 610, and, likewise, 

15 determines the position and orientation of the master by means of the master frame 622 
relative to the eye frame 612. 

As mentioned, the surgeon grips the master by locating his or her thumb 
and index finger over the pincher formation 706. When the surgeon's thumb and index 
finger are located on the pincher formation, the point of intersection 3 A is positioned 

20 inwardly of the thumb and index finger tips. The master frame having its origin at 3A is 
effectively mapped onto the end effector frame 618, having its origin at the pivotal 
connection 60 of the end effector 58 as viewed by the surgeon in the viewer 202. Thus, 
when performing the surgical procedure, and the surgeon manipulates the position and 
orientation of the pincher formation 706 to cause the position and orientation of the end 

25 effector 58 to follow, it appears to the surgeon that his or her thumb and index finger are 
mapped onto the fingers of the end effector 58 and that the pivotal connection 60 of the 
end effector 58 corresponds with a virtual pivot point of the surgeon's thumb and index 
finger inwardly from the tips of the thumb and index finger. It will be appreciated that 
depending upon the actual configuration of the pincher formation, in particular the point 

30 of intersection of the axes 1 and 3 relative to the position of the pincher formation 706, 
the frame 622 on the master 700 can be offset from the intersection 3 A so as to approach 
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a point relative to the surgeon's hand at which point the pivotal connection 60 
approximately corresponds. 

Accordingly, as the surgical procedure is being performed the position and 
orientation of the fingers of the end effector tracks orientation and position changes of the 
5 surgeon's thumb and index finger in a natural intuitive or superimposed fashion. 

Furthermore, actuation of the end effector 58, namely causing the end effector fingers 
selectively to open and close, corresponds intuitively to the opening and closing of the 
surgeon's thumb and index finger. Thus, actuation of the end effector 58 as viewed in the 
viewer 302 is performed by the surgeon in a natural intuitive manner, since the pivot 

10 point 60 of the end effector 58 is appropriately mapped onto a virtual pivot point between 
the surgeon's thumb and index finger. 

It will be appreciated that the end effector frame 618 can, where 
appropriate, be offset relative to the pivotal connection 60. Thus, for example, should the 
end effector (as shown in the display) have fingers of a relatively long length, the origin 

15 of the end effector frame can be offset in a direction toward the end effector finger tips. It 
will also be appreciated that using positional and/or orientational offsets between the 
master frame 622 and the intersection 3 A, as well as between the end effector frame 618 
and the pivotal connection 60, the mapping of the pincher formation 706 onto the end 
effector 58 may be shifted, for example to map the tips of the pincher formation onto the 

20 tips of the end effector. These alternative mappings are illustrated in Figure 1 1 A. 

Generally, a first pincher element 706A will preferably be substantially 
connected to a first end effector element 58.1, while a second pincher element 706B is 
substantially connected to a second end effector element 58.2. Optionally, point 3A 
(which is ideally near the center of rotation of the gimbal structure of master 700, 706A, 

25 and 706B), adjacent the pivotal connection between the pincher elements, may be 

substantially connected with pivotal connection 60 on the slave. This also effectively 
provides a substantial connection between the pivot point on the surgeon's hand H and 
pivotal connection 60, as the surgeon will often grip the master with the hand's pivot point 
(at the base of the surgeon's finger and thumb) disposed along the pivot point of the 

30 pincher. Alternatively, midpoint MP1 disposed between the tips of the pincher elements 
may be substantially connected to midpoint MP2 disposed between the tips of the end 
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effector elements. Each of the higher levels of connection described herein may 
optionally be provided by this mapping. 

Figures 1 IB and C more clearly illustrate corresponding mapping points 
between the handle of the master controller and end effector of the slave, while 
5 Figure 1 1C schematically illustrates method steps for selecting these corresponding 
mapping points. In general, interchangeable end effectors having different end effector 
element lengths may be accommodated by varying the mapping point of the handle or the 
end effector. Such variation in mapping points may also be used when the magnification 
of the image shown at the display changes significantly. For example, substantial 

10 connection of pivotal connection 60 of the end effector and intersection 3 A of the handle 
may be appropriate when the end effector is shown at a first magnification, but may be 
inappropriate when magnification of the end effector is increased significantly, or when 
an alternative end effector having longer end effector elements is attached to the slave. In 
either circumstance, it may be appropriate to alter the master/slave interaction to 

15 substantially connect midpoint MP2 of the master to midpoint MPV of the end effector, 
as illustrated in Figure 1 IB. 

As a preliminary matter, it is beneficial in robotic surgery systems to 
provide a master controller having a gimbal point GP adjacent the handle to be gripped by 
the surgeon. This avoids large master inertia when the surgeon rapidly rotates the handle, 

20 as often occurs during surgical procedures. By having a master which has multiple 

degrees of freedom intersecting at the gimbal point GP (ideally having three orientational 
degrees of freedom intersecting at the gimbal point), and by having the gimbal point 
coincident with the handle, inertia of rapid rotational movements at the master can be 
quite low. 

25 As described above, it is often beneficial to coordinate movements of the 

slave so that an image of pivotal connection 60 of the slave appears substantially 
connected to pincher formation pivot point 3A between the pincher or grip elements 
706A, 706B. However, when end effector elements 58.1, 58.2 extend a considerable 
distance beyond pivotal connection 60 (as shown in the display adjacent the master 

30 controller), the surgeon may feel that manipulation of these long end effector elements 
from the distant pivotal connection becomes awkward. Similarly, when manipulating a 
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single end effector element such as a scalpel which is much longer (as displayed at the 
master control station) than the master handle, the surgeon may be given the impression 
of cutting with a long-handled sword, rather than an easily controlled scalpel As 
described above, one alternative to overcome an awkward disparity in grip/end effector 
5 lengths is to map the surgical workspace and master controller workspace together so that 
the midpoints MP1, MP2 between the end effector jaw ends and the handle grip member 
ends are substantially connected. By mapping the surgical and master workspace so that 
these midpoints are substantially connected, the surgeon can coordinate movement using 
the end effector despite significant differences in length between the end effector 

10 elements and the grip elements. 

The mapping point need not be limited to any particular point. In the 
exemplary embodiment, a middle axis of the grip members MAG is generally defined 
midway between pincher elements 706A, 706B, while a similar middle axis of the end 
effector MAE is defined midway between the end effector elements 58.1, 58.2. The 

15 mapping point (or point of substantial connection) of the master will preferably be 
disposed along gripping middle axis MAG, ideally in a range from intersection 3A to 
midpoint MP2. Similarly, the mapping or substantial connection point of the end effector 
will preferably be disposed along middle axis MAE, ideally in a range from pivotal 
connection 60 to midpoint MP1. 

20 Figure 1 1C schematically illustrates a method for determining the location 

of the substantially connected mapping points along the handle and end effector. First, 
the location of the surgeon's hand along the master handle is reviewed to determine the 
position of the surgeon's fingers relative to the gimbal point GP. In one embodiment, the 
offset distance between a location of the surgeon's fingertips and the gimbal point GP 

25 defines an offset distance. This offset distance is scaled using a scaling factor, typically 
using a ratio between a length of the grip members and the length of the end effector 
elements, a magnification of the display, or the like. For example, using numbers typical 
of the exemplary robotic surgery system, the offset distance is scaled by multiplying it by 
one-third, as the grip members typically have a length of about three times the end 

30 effector element lengths. This scaling factor may change with tool changes (when end 
effectors having longer or shorter end effector elements are used), or the like. The 
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location of the mapping points on the slave can then be calculated, for example, at a 
position offset from midpoint MP1 toward pivotal connection 60 along the end effector 
middle axis MAE by the scaled offset distance. This mapping point of the end effector 
may then be substantially connected to gimbal point GP of the master. 
5 It will be appreciated that the cart frame 624 can be chosen at any 

convenient location in which its origin corresponds with a location on the cart 300 which 
does not vary relative to its base 99. The surgeon's station frame 634 can likewise be 
chosen at any convenient location such that its origin is located at a position which does 
not vary relative to a base 642 thereof. Furthermore, to determine the position and 

10 orientation of the camera frame 610 relative to the cart frame 624, use can be made of a 
plurality of different intermediate frame paths. To determine the position and orientation 
of the end effector frame 618 relative to the cart frame 624 use can also be made of a 
plurality of different intermediate frame paths. 

However, it has been found that should the intermediate frame paths be 

15 appropriately selected, the control system i3 then arranged to be readily adaptable to 

accommodate modular replacement of modular parts having different characteristics than 
the modular parts being replaced. It will be appreciated that selecting intermediate frames 
also eases the computational process involved in determining master and slave position 
and orientation. 

20 Referring again to Figure 10 of the drawings, the cart frame is chosen at 

624, as already mentioned. It will be appreciated that determining the position of the 
fulcrum frame 630 relative to the cart frame 624 is achieved through appropriately 
positioned sensors, such as potentiometers, encoders, or the like. Conveniently, the 
fulcrum frame position 630 relative to the cart frame 624 is determined through two 

25 intermediate frames. One of the frames is a carriage guide frame 644 which has its origin 
at a convenient location on a guide along which the carriage 97 is guided. The other 
frame, an arm platform frame indicated at 646 is positioned at an end of the setup joint 
arm 95 on which the robotic arm 12 is mounted. Thus, when slave position and 
orientation is determined relative to the cart frame 624, the carriage guide frame 644 

30 position relative to the cart frame 624 is determined, then the platform frame 646 position 
relative to the carriage guide frame 644, then the fulcrum frame 630 relative to the 
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platform frame 646, and then the slave orientation and position relative to the fulcrum 
frame 630, thereby to determine the slave position and orientation relative to the cart 
frame 624. It will be appreciated that the slave position and orientation relative to the cart 
frame 624 is determined in this manner for each arm 10 and in similar fashion for the 
5 camera frame 610, through its arm 302, relative to the cart frame 624. 

Referring to Figure 1 1, the position and orientation of the master control is 
determined by determining the position of a base frame 648 relative to the surgeon's 
station frame 634, then determining the position of the platform frame 640 relative to the 
base frame 648, and then determining master position and orientation relative to the 
10 platform frame 640. The position and orientation of the master frame 622 relative to the 
surgeon's station frame 634 is then readily determined through routine calculation using 
trigonometric relationships. It will be appreciated that the position and orientation of the 
other master frame relative to the surgeon console frame 634 is determined in a similar 
fashion. 

15 Referring to Figure 10, by choosing the frames as described, the setup joint 

95 can be replaced with another setup joint while the same robotic arm is used. The 
control system can then be programmed with information, e.g., arm lengths and/or the 
like, relating to the new setup joint only. Similarly, the robotic arm 10 can be replaced 
with another arm, the control system then requiring programming with information, e.g., 

20 fulcrum position and/or the like, relating to the new robotic arm only. It will be 

appreciated that in this way the endoscope arm 302 and its associated setup joint can also 
be independently replaced, the control system then requiring programming of information 
relating only to the part being replaced. Furthermore, referring to Figure 1 1, the setup 
joint and master control can also independently be replaced, the control system requiring 

25 programming of information relating to the characteristics of the new part only. 

Figure 12 schematically illustrates a high level control architecture for a 
master/slave robotic system 1000. Beginning at the operator input, a surgeon 1002 moves 
an input device of a master manipulator 1004 by applying manual or human forces fn 
against the input device. Encoders of master manipulator 1004 generate master encoder 

30 signals e m which are interpreted by a master input/output processor 1006 to determine the 
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master joint positions 0 m . The master joint positions are used to generate Cartesian 
positions of the input device of the mastery using a master kinematics model 1008. 

Starting now with the input from the surgical environment 1018, the tissue 
structures in the surgical workspace will impose forces f e against a surgical end effector 
5 (and possibly against other elements of the tool and/or manipulator). Environmental 
forces f e from the surgical environment 1018 alter position of the slave 1016, thereby 
altering slave encoder values e 5 transmitted to the slave input/output processor 1014. 
Slave input/output processor 1014 interprets the slave encoder values to determine joint 
positions 9 S , which are then used to generate Cartesian slave position signals x s according 
10 to the slave kinematics processing block 1012. 

The master and slave Cartesian positions x m , x s are input into bilateral 
controller 1010, which uses these inputs to generate the desired Cartesian forces to be 
applied by the slave f s so that the surgeon can manipulate the salve as desired to perform a 

* 

surgical procedure. Additionally, bilateral controller 1010 uses the Cartesian master and 

15 slave positions x my x, to generate the desired Cartesian forces to be applied by the master 
f m so as to provide force feedback to the surgeon. 

In general, bilateral controller 1010 will generate the slave and master 
forces fs>f m by mapping the Cartesian position of the master in the master controller 
workspace with the Cartesian position of the end effector in the surgical workspace 

20 according to a transformation. Preferably, the control system 1000 will derive the 

transformation in response to state variable signals provided from the imaging system so 
that an image of the end effector in a display appears substantially connected to the input 
device. These state variables will generally indicate the Cartesian position of the field of 
view from the image capture device, as supplied by the slave manipulators supporting the 

25 image capture device. Hence, coupling of the image capture manipulator and slave end 
effector manipulator is beneficial for deriving this transformation. Clearly, bilateral 
controller 1010 may be used to control more than one slave arm, and/or may be provided 
with additional inputs. 

Based generally on the difference in position between the master and the 

30 slave in the mapped workspace, bilateral controller 1010 generates Cartesian slave forced 
to urge the slave to follow the position of the master. The slave kinematics 1012 are used 
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to interpret the Cartesian slave forces f s to generate joint torques of the slave x s which will 
result in the desired forces at the end effector. Slave input/output processor 1014 uses 
these joint torques to calculate slave motor currents i Sj which reposition the slave x e 
within the surgical worksite. 
5 The desired feedback forces from bilateral controller are similarly 

interpreted from Cartesian force on the master^ based on the master kinematics 1008 to 
generate master joint torques x s . The master joint torques are interpreted by the master 
input/output controller 1006 to provide master motor current i m to the master manipulator 
1004, which changes the position of the hand held input device x h in the surgeon's hand. 

1 0 It will be recognized that the control system 1 000 illustrated in Figure 1 2 

is a simplification. For example, the surgeon does not only apply forces against the 
master input device, but also moves the handle within the master workspace. Similarly, 
the motor current supplied to the motors of the master manipulator may not result in 
movement if the surgeon maintains the position of the master controller. Nonetheless, the 

15 motor currents do result in tactile force feedback to the surgeon based on the forces 
applied to the slave by the surgical environment. Additionally, while Cartesian 
coordinate mapping is preferred, the use of spherical, cylindrical, or other reference 
frames may provide at least some of the advantages of the invention. 

Further aspects of the control system of the minimally invasive surgical 

20 apparatus will now be described with reference to Figure 12 A. 

Figure 12A indicates the control steps whereby the control system of the 
minimally invasive surgical apparatus determines slave position and orientation, namely 
the position and orientation of the end effector frame 618 in the camera frame 610. 

The position or offsets of the carriage guide frame 644 relative to the cart 

25 frame 624 is indicated at 621. The offsets at 621 are fed through a forward kinematics 
block (FKIN) at 623 to yield corresponding Cartesian coordinates of the frame 644 
relative to the cart frame 624. 

Sensors 625 operatively associated with the setup joint arm 95 and sensors 
determining the height of the carriage 97, are read by a processor 627 to determine 

30 translational and joint positions. The translational and joint positions are then input to an 
FKIN block 629 to determine corresponding Cartesian coordinates. At 631, the Cartesian 
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coordinates of the carriage guide frame 644 relative to the cart frame 624 and the 
Cartesian coordinates of the platform frame 646 relative to the carriage frame 644 are 
used to determine the Cartesian coordinates of the platform frame 646 relative to the cart 
frame 624. 

5 Since the position of the fulcrum 49 relative to the platform frame 646 

does not change, an offset relative to the platform frame 646, indicated at 633, is input to 
an FKIN controller at 635 to yield Cartesian coordinates of the fulcrum frame 630 relative 
to the platform frame 646. It will be appreciated that, where appropriate, the term FKIN 
controller is to be interpreted to include an appropriate conversion matrix and kinematic 
10 relationships. At 637, the Cartesian coordinates of the fulcrum frame 630 relative to the 
cart frame 624 are determined by means of the values determined at 631 and 635 
respectively. 

It will be appreciated that, in similar fashion, the Cartesian coordinates of 
the fulcrum of the endoscope is determined relative to the cart frame 624. This is 
15 indicated at 639. 

As mentioned, the position and orientation of the endoscope 304 can be 
varied. The position and orientation of the endoscope 304 can be varied during set up of 
the cart 300 before the surgical procedure commences or during the performance of a 
surgical procedure should the surgeon wish to view the surgical site from a different 
20 location. 

To enable the control system to determine endoscope position and 
orientation relative to the cart frame 624, sensors are provided on its associated arm 302. 
These sensors, indicated at 641, are read by a processor at 643 to determine joint 
positions. The joint positions thus determined are fed to an FKIN controller at 645, 

25 together with the Cartesian coordinates determined at 639 to determine endoscope 

orientation and position relative to the cart frame 624. These values are then input to 647 
together with the values determined at 637, so as to enable the fulcrum frame 630 of the 
slave to be determined relative to the camera frame 610. 

During the course of the surgical procedure, the slave orientation and 

30 position is normally constantly changing. Varying joint positions and velocities are fed 
into an FKIN controller at 653, together with the Cartesian coordinate values of the slave 
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position relative to the camera frame determined at 647 to yield Cartesian position and 
velocity of the slave, namely the end effector frame 618, relative to the camera frame 610, 
as indicated by arrows 655, 657 respectively. For economy of words, Cartesian position 
is to be interpreted to include Cartesian orientation in the rest of this specification where 
5 appropriate. The varying joint positions and velocities are fed into the FKIN block 653 
from a simulation domain as described in greater detail hereinbelow. 

Referring now to Figure 13, master position and orientation relative to the 
viewer frame 612 will now be described. 

The base frame 648 normally does not change relative to the surgeon 

10 station frame 634. Similarly, the frame at 640 normally does not change relative to the 
base frame 648. As mentioned, setup joints can optionally be provided at 632 if desired. 
For the sake of the description which follows, the position of the frame at 640 relative to 
the base frame 648 is assumed to be unchangeable. Naturally, if setup joint arms are 
provided at 632, appropriate sensors would then be provided to enable the position of the 

15 frame at 640 to be determined relative to the frame at 648. 

Referring now to Figure 13, offsets determining the frame 648 position 
relative to the surgeon station frame 634, as indicated at 659, are fed through an FKIN 
controller 665 to yield Cartesian coordinates of the base frame 648 relative to the surgeon 
station frame 634. Similarly, offsets relating to frame 640 position relative to base frame 

20 648 position, as indicated at 661, are fed through an FKIN controller at 663 to yield 
Cartesian coordinates of the frame 640 relative to the base frame 648. From the values 
derived at 665, 663, the Cartesian coordinates of the frame 640 relative to the surgeon 
station frame 634 are determined at 667. 

Offsets at 697 relating to a viewer base frame, not indicated in Figure 1 1, 

25 are fed through an FKIN controller at 669 to yield corresponding Cartesian coordinates of 
the base frame relative to the frame 634. The viewer 202 can be positionally adjustable 
relative to the rest of the operator station 200. To enable a viewer position relative to the 
viewer base frame to be determined, appropriately positioned sensors 671 are provided. 
Sensor readings from these sensors at 671 are processed at 673 to determine joint or 

30 translational positions which are then fed through an FKIN controller at 675 to yield 

Cartesian coordinates of the viewer frame relative to the viewer base frame. At 677, the 
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viewer frame position in Cartesian coordinates relative to the surgeon station frame 634 
are determined from the values derived at 669 and 675 respectively. 

Offsets corresponding to the position of the surgeon's eyes relative to the 
viewer frame at 679 are fed through an FKIN controller at 681 to yield Cartesian 
5 coordinates of the position of the surgeon's eyes relative to the viewer frame. At 683, the 
values from 677 and 681 are used to determine the surgeon's eye frame 612 relative to the 
surgeon station frame 634. 

At 685, the values from 667 and 683 are used to determine the position of 
the frame 640 relative to the eye frame 612. 

1 0 Naturally, master position and orientation relative to the eye frame 612 is 

continually changing during the course of a surgical procedure. The sensors on the 
master 700, indicated at 687, are read by a processor at 689 to determine master joint 
position and velocity. These joint position and velocity values are then fed through an 
FKIN controller at 691, together with the value derived at 685 to yield master Cartesian 

15 position and velocity values 693, 695 relating to Cartesian position and velocity of master 
frame 622, relative to the eye frame 612. 

At the commencement of a surgical procedure, an initial position of the 
master 700 is set to correspond with an initial position of the slave. Thereafter, as the 
master 700 is moved, the control system monitors such movement and commands the 

20 slave to track the master movement. Thus, at the commencement of a surgical procedure, 
the frame 618 on the slave at the pivotal connection 60, relative to its reference frame 610 
at the viewing end 306 of the endoscope 304, at the initial position, is mapped onto the 
master frame 622 relative to its reference eye frame 612 at its initial position. Similarly, 
the system maps an initial orientation of the pincher formation frame 622 with an initial 

25 orientation of the end effector frame 618. Thus, the orientation of the end effector frame 
61 8 is also caused to track the orientation of the master frame 622. The position and 
orientation of the slave in the camera frame 610 need not correspond identically with the 
position and orientation of the master in the eye frame 612. Accordingly, offsets can be 
introduced relating to the orientation and the position of the end effector frame 618 

30 relative to the camera frame 61 0 to define an arbitrary end effector frame position and 
orientation which corresponds to a master frame 622 position and orientation in the eye 
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frame 612. It will be appreciated that the control system can readily determine the 
orientation and the position of the end effector frame 618 relative to the camera frame 
610 at which it is to correspond with that of the master frame relative to the eye frame by 
means of the frames and offsets discussed above. Thus, even during the course of a 
5 surgical procedure, if the control between master and slave is interrupted and the 

endoscope is moved, or one or both of the surgical instruments are repositioned through 
different ports of entry, or the master positions are changed at the surgeon's console, or 
the like, re-mapping of slave relative to master in their respective camera and eye frames 
can readily be achieved by the control system. 
10 The control system, generally indicated by reference numeral 810, will 

now be described in greater detail with reference to Figure 14 of the drawings, in which 
like reference numerals are used to designate similar parts or aspects, unless otherwise 
stated. 

As mentioned earlier, the master control 700 has sensors, e.g., encoders, or 

15 potentiometers, or the like, associated therewith to enable the control system 810 to 
determine the position of the master control 700 in joint space as it is moved from one 
position to a next position on a continual basis during the course of performing a surgical 
procedure. In Figure 14, signals from these positional sensors are indicated by arrow 814. 
Positional readings measured by the sensors at 687 are read by the processor indicated at 

20 689 (refer to Figure 13). It will be appreciated that since the master control 700 includes 
a plurality of joints connecting one arm member thereof to the next, sufficient positional 
sensors are provided on the master 700 to enable the angular position of each arm 
member relative to the arm member to which it is joined to be determined thereby to 
enable the position and orientation of the master frame 622 on the master to be 

25 determined. As the angular positions of one arm member relative to the arm member to 
which it is joined is read cyclically by the processor 689 in response to movements 
induced on the master control 700 by the surgeon, the angular positions are continuously 
changing. The processor at 689 reads these angular positions and computes the rate at 
which these angular positions are changing. Thus, the processor 689 reads angular 

30 positions and computes the rate of angular change, or joint velocity, on a continual basis 
corresponding to the system processing cycle time, i.e., 1300 Hz. Joint position and joint 
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velocity commands thus computed at 689 are then input to the Forward Kinematics 
(FKIN) controller at 691, as already described hereinabove. 

At the FKIN controller 691, the positions and velocities in joint space are 
transformed into corresponding positions and velocities in Cartesian space, relative to the 
5 eye frame 612 (refer to Figures 1 1 and 13). The FKIN controller 691 is a processor 
which typically employs a Jacobian (J) matrix to accomplish this. It will be appreciated 
that the Jacobian matrix transforms angular positions and velocities into corresponding 
positions and velocities in Cartesian space by means of conventional trigonometric 
relationships. Thus, corresponding positions and velocities in Cartesian space, or 

10 Cartesian velocity and position commands, are computed by the FKIN controller 691 
which correspond to Cartesian position and velocity changes of the master frame 622 in 
the eye frame 612. 

The velocity and the position in Cartesian space is input into a Cartesian 
controller, indicated at 820, and into a scale and offset converter, indicated at 822. 

15 The minimally invasive surgical apparatus provides for a scale change 

between master control input movement and responsive slave output movement. Thus, a 
scale can be selected where, for example, a 1-inch movement of the master control 700 is 
transformed into a corresponding responsive 1/5-inch movement on the slave. At the 
scale and offset step 822, the Cartesian position and velocity values are scaled in 

20 accordance with the scale selected to perform the surgical procedure. Naturally, if a scale 
of 1:1 has been selected, no change in scale is effected at 822. Similarly, offsets are taken 
into account which determine the corresponding position and/or orientation of the end 
effector frame 618 in the camera frame 610 relative to the position and orientation of the 
master frame 622 in the eye frame 612. 

25 After a scale and offset step is performed at 822, a resultant desired slave 

position and desired slave velocity in Cartesian space is input to a simulated or virtual 
domain at 812, as indicated by arrows 811. It will be appreciated that the labeling of the 
block 812 as a simulated or virtual domain is for identification only. Accordingly, the 
simulated control described hereinbelow is performed by elements outside the block 812 

30 also. 
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The simulated domain 812 will be described in greater detail hereinbelow. 
However, the steps imposed on the desired slave velocity and position in the virtual 
domain 812 will now be described broadly for ease of understanding of the description 
which follows. A current slave position and velocity is continually monitored in the 
5 virtual or simulated domain 812. The desired slave position and velocity is compared 
with the current slave position and velocity. Should the desired slave position and/or 
velocity as input from 822 not cause transgression of limitations, e.g., velocity and/or 
position and/or singularity, and/or the like, as set in the virtual domain 812, a similar 
Cartesian slave velocity and position is output from the virtual domain 812 and input into 

10 an inverse scale and offset converter as indicated at 826. The similar velocity and 

position output in Cartesian space from the virtual domain 812 is indicated by arrows 813 
and corresponds with actual commands in joint space output from the virtual domain 812 
as indicated by arrows 815 as will be described in greater detail hereinbelow. From the 
inverse scale and offset converter 826, which performs the scale and offset step of 822 in 

15 reverse, the reverted Cartesian position and velocity is input into the Cartesian controller 
at 820. At the Cartesian controller 820, the original Cartesian position and velocities as 
output from the FKIN controller 691 is compared with the Cartesian position and velocity 
input from the simulated domain 812. If no limitations were transgressed in the simulated 
domain 812 the velocity and position values input from the FKIN controller 691 would be 

20 the same as the velocity and position values input from the simulated domain 812. In 
such a case, a zero error signal is generated by the Cartesian controller 820. 

In the event that the desired Cartesian slave position and velocity input at 
811 would transgress one or more set limitations, the desired values are restricted to stay 
within the bounds of the limitations. Consequently, the Cartesian velocity and position 

25 forwarded from the simulated domain 812 to the Cartesian controller 820 would then not 
be the same as the values from the FKIN controller 691 . In such a case, when the values 
are compared by the Cartesian controller 820, an error signal is generated. 

The type of limitations imposed on the desired slave Cartesian position 
and velocity will be described in greater detail hereinbelow. 

30 Assuming that a zero error is generated at the Cartesian controller 820 no 

signal is passed from the Cartesian controller or converter 820. In the case that an error 
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signal is generated the signal is passed through a summation junction 827 to a master 
transpose kinematics controller 828. 

The error signal is typically used to calculate a Cartesian force. The 
Cartesian force is typically calculated, by way of example, in accordance with the 

■ 

5 following formula: 

Fcart = K(A x) + B(A^r) 

where K is a spring constant, B is a damping constant, A jt is the difference between the 
Cartesian velocity inputs to the Cartesian controller 820 and A jc is the difference between 

■ 

the Cartesian position inputs to the Cartesian controller 820. It will be appreciated that 

10 for an orientational error, a corresponding torque in Cartesian space is determined in 
accordance with conventional methods. 

The Cartesian force corresponds to an amount by which the desired slave 
position and/or velocity extends beyond the limitations imposed in the simulated domain 
812. The Cartesian force, which could result from a velocity limitation, a positional 

15 limitation, and/or a singularity limitation, as described in greater detail below, is then 

converted into a corresponding torque signal by means of the master transpose kinematics 
controller 828 which typically includes a processor employing a Jacobian Transpose (J T ) 
matrix and kinematic relationships to convert the Cartesian force to a corresponding 
torque in joint space. The torque thus determined is then input to a processor at 830 

20 whereby appropriate electrical currents to the motors associated with the master 700 are 
computed and supplied to the motors. These torques are then applied on the motors 
operatively associated with the master control 700. The effect of this is that the surgeon 
experiences a resistance on the master control to either move it at the rate at which he or 
she is urging the master control to move, or to move it into the position into which he or 

25 she is urging the master control to move. The resistance to movement on the master 

control is due to the torque on the motors operatively associated therewith. Accordingly, 
the higher the force applied on the master control to urge the master control to move to a 
position beyond the imposed limitation, the higher the magnitude of the error signal and 
the higher an opposing torque on the motors resisting displacement of the master control 

30 in the direction of that force. Similarly, the higher the velocity imposed on the master 
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beyond the velocity limitation, the higher the error signal and the higher the opposing 
torque on the motors associated with the master. 

The imposition of the limitations in the simulated domain 812 will now be 
described in greater detail with reference to Figure 1 5 of the drawings. In Figure 15, like 
5 reference numerals are used to designate similar parts or aspects, unless otherwise stated. 

The slave desired Cartesian velocity is passed from the scale and offset 
converter 822 through a summation junction at 832. It will be appreciated that the slave 
desired Cartesian velocity is passed through the summation junction 832 sequentially at 
the rate of the control system processing cycle, namely 1300 Hz. At the junction 832, an 
10 error signal is imparted on the slave desired Cartesian velocity when the desired velocity 
of a prior desired Cartesian velocity signal would have instructed the simulated slave to 
transgress one or more limitations. This will be described in greater detail hereinbelow. 
If the prior desired slave velocity would not have caused a transgression, no error signal 
would have been generated and the desired slave velocity would then pass through the 
15 summation junction 832 unchanged. The velocity signal passed from the summation 
junction 832 is referred to as Cartesian reference velocity as indicated by arrow 833. 

From the summation junction 832, the Cartesian reference velocity is fed 
to a simulation block 834. The reference velocity is then compared with the limitations in 
the simulation block 834, as will be described in greater detail hereinbelow with reference 
20 to Figures 16 to 21 of the drawings. 

In the case where the slave reference velocity does not transgress a 
limitation, the slave reference velocity passes through the simulation block 834 
unchanged. However, a corresponding simulated slave joint velocity is computed in the 
simulation block 834. 

25 The simulated joint velocity is integrated in the simulation block 834 to 

yield a corresponding simulated joint position. The simulated joint velocity and position 
is output from the simulation block 834 as indicated by arrows 835. 

The simulated joint velocity and position is then passed through a filter at 
838. The filter 838 is arranged to separate tremors from the velocity and position signals. 

30 It will be appreciated that such tremors could result from inadvertent shaking of the 

master control which can be induced on the master control by the surgeon. Since it would 
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be desirable to remove such tremor movements from the actual slave velocity and 
position signals so as to enhance slave precisional movement in response to master input, 
these tremors are filtered from the velocity and position signals by means of the filter 838. 
After the filtering step at 838, resultant slave joint velocity and position signals are passed 
5 to the slave as indicated by arrows 815 and as will be described in greater detail 

hereinbelow. It will be appreciated that the simulated slave joint position and/or velocity 
signal can be modified in any desired manner at 838. Typically, modifications not 
requiring feedback to the master can be implemented at 838. Thus, the filtering step 838 
is not necessarily limited to filtering tremors from the signal only. Jn addition, or instead, 

10 the frequency of the position and/or velocity signals may be modified to inhibit resonance 
in the slave, for example. 

Still referring to Figure 15 of the drawings, the simulated joint velocity 
and position, after passing through the simulation block 834, is routed through an FKIN 
controller at 653 to compute corresponding velocities and positions in Cartesian space, as 

15 described with reference to Figure 12A of the drawings. The signals are then passed to 
the Cartesian controller 820 as already described with reference to Figure 14. 

Still referring to Figure 15, the position signal from the FKIN controller 
653 is routed into a Cartesian scaled error block at 844. The desired Cartesian slave 
position derived from the scale and offset block 822 is also routed into the Cartesian 

20 scaled error block 844., The two signals are compared at 844 to compute an error signal 
should they not correspond. Should the two signals be equal, namely where the desired 
slave velocity signal was not restricted in the simulated domain 834, no error signal is 
generated. 

In the case where the desired slave velocity was restricted in the simulation 
25 block 834, the simulated joint velocity output would not correspond with the reference 
Cartesian slave velocity input to the simulation block 834. Accordingly, after integration 
in the simulation block 834, and conversion to Cartesian space by the FKIN controller 
653, the resultant corresponding Cartesian position would not correspond with the 
original desired Cartesian slave position input to the Cartesian scaled error block 844. 
30 Accordingly, an error signal of a magnitude determined typically by subtraction of the 
resultant Cartesian position from the original desired position and multiplication with an 
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appropriate constant, is generated by the Cartesian scaled error block 844. This error 
signal is imposed on the next desired slave velocity signal at the summation junction 832. 

It will be appreciated that only the velocity signal is input to the simulation 
block 834. Thus, limitations are imposed in a dynamic fashion in the simulation block. 
5 The simulated slave position does not necessarily track the master position 

simultaneously. This is particularly the case where a limitation has been imposed in the 
simulation block 834. For example, should a velocity limit have been imposed where the * 
master was moved too quickly, a degree of lagging of the simulated slave position to 
catch up with the master position results. Accordingly, a discrepancy between the master 

10 and the slave positions ensues. By means of the positional error generated at 844, an 
appropriate velocity signal change is effected at the junction 852 to effect a positional 
"catch up" function on the velocity signal. Thus, should the master be brought to rest 
where a positional error is generated, the velocity signal input to 832 would be zero, but a 
Cartesian reference velocity would still be input to the simulation block 834 to effect the 

15 catching up of the simulated slave position with that of the master. 

Referring once again to Figure 14 of the drawings, the resultant slave joint 
velocity and position signal is passed from the simulated domain 812 to a joint controller 
848. At the joint controller 848, the resultant joint velocity and position signal is 
compared with the current joint position and velocity. The current joint position and 

20 velocity is derived through the sensors on the slave as indicated at 849 after having been 
processed at an input processor 85 1 to yield slave current position and velocity in joint 
space. 

The joint controller 848 computes the torques desired on the slave motors 
to cause the slave to follow the resultant joint position and velocity signal taking its 
25 current joint position and velocity into account. The joint torques so determined are then 
routed to a feedback processor at 852 and to an output processor at 854. 

The joint torques are typically computed, by way of example, by means of 
the following formula: 

T = K(A#) + B(A#) 
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where K is a spring constant, B is a damping constant, A 0 is the difference between the 
joint velocity inputs to the joint controller 851, and A 0 is the difference between the joint 

> 

position inputs to the joint controller 851. . 

The output processor 854 determines the electrical currents to be supplied 
5 to the motors associated with the slave to yield the commanded torques and causes the 
currents to be supplied to the motors as indicated by arrow 855. 

From the feedback processor 852 force feedback is supplied to the master. 
As mentioned earlier, force feedback is provided on the master 700 whenever a limitation 
is induced in the simulated domain 812. Through the feedback processor 852 force 

10 feedback is provided directly from the slave 798, in other words, not through a virtual or 
simulated domain but through direct slave movement. This will be described in greater 
detail hereinbelow. 

As mentioned earlier, the slave indicated at 798 is provided with a 
plurality of sensors. These sensors are typically operatively connected to pivotal joints on 

15 the robotic arm 10 and on the instrument 14. 

These sensors are operatively linked to the processor at 85 1 . It will be 
appreciated that these sensors determine current slave position. Should the slave 798 be 
subjected to an external force great enough to induce reactive movement on the slave 798, 
the sensors will naturally detect such movement. Such an external force could originate 

20 from a variety of sources such as when the robotic arm 1 0 is accidentally knocked, or 
knocks into the other robotic arm 10 or the endoscope arm 302, or the like. As 
mentioned, the joint controller 848 computes torques desired to cause the slave 798 to 
follow the master 700. An external force on the slave 798 which causes its current 
position to vary also causes the desired slave movement to follow the master to vary. 

25 Thus a compounded joint torque is generated by the joint controller 848, which torque 
includes the torque desired to move the slave to follow the master and the torque desired 
to compensate for the reactive motion induced on the slave by the external force. The 
torque generated by the joint controller 848 is routed to the feedback processor at 852, as 
already mentioned. The feedback processor 852 analyzes the torque signal from the joint 

30 controller 848 and accentuates that part of the torque signal resulting from the extraneous 
force on the slave 798. The part of the torque signal accentuated can be chosen 
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depending on requirements. In this case, only the part of the torque signal relating to the 
robotic arm 12, 12, 302 joints are accentuated. The torque signal, after having been 
processed in this way is routed to a kinematic mapping block 860 from which a 
corresponding Cartesian force is determined. At the kinematic block 860, the information 
5 determining slave fulcrum position relative to the camera frame is input from 647 as 

indicated. In this regard refer to Figure 12A of the drawings. Thus, the Cartesian force is 
readily determined relative to the camera frame. This Cartesian force is then passed 
through a gain step at 862 appropriately to vary the magnitude of the Cartesian force. 
The resultant force in Cartesian space is then passed to the summation junction at 827 and 

10 is then communicated to the master control 700 as described earlier. 

Reference numeral 866 generally indicates another direct force feedback 
path of the control system 810, whereby direct force feedback is supplied to the master 
control 700. The path 866 includes one or more sensors which are not necessarily 
operatively connected to slave joints. These sensors can typically be in the form offeree 

15 or pressure sensors appropriately positioned on the surgical instrument 14, typically on 
the end effector 58. Thus, should the end effector 58 contact an extraneous body, such as 
body tissue at the surgical site, it generates a corresponding signal proportionate to the 
force of contact. This signal is processed by a processor at 868 to yield a corresponding 
torque. This torque is passed to a kinematic mapping block 864, together with 

20 information from 647 to yield a corresponding Cartesian force relative to the camera 
frame. From 864, the resultant force is passed through a gain block at 870 and then 
forwarded to the summation junction 827. Feedback is imparted on the master control 
700 by means of torque supplied to the motors operatively associated with the master 
control 700 as described earlier. It will be appreciated that this can be achieved by means 

25 of any appropriate sensors such as current sensors, pressure sensors, accelerometers, 
proximity detecting sensors, or the like. 

In some embodiments, resultant forces from kinematic mapping 864 may 
be transmitted to an alternative presentation block 864. 1 so as to indicate the applied 
forces in an alternative format to the surgeon. For example, the total force may be 

30 presented in the form of a bar graph shown on the display, typically beyond a border of 
the displayed image from the image capture device. Alternatively, the resulting forces 
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applied against the slave may be graphically shown as a force vector, either outside the 
image border on the display, or overlaid over the slave structure in the displayed image. 
Still further presentation alternatives are possible, including the use of false colors (for 
example, changing the color of a slave component to yellow and then red as the 
5 component approaches and then reaches its maximum force capability), or audibly 
indicating the force on the slave structure with a tone which increases in pitch and/or 
volume as forces increase. Additional tactile representations of force may be employed, 
for example, using heat to indicate force or an inertial actuator which, for example, 
vibrates with increasing speed or amplitude as forces increase. Such inertial actuators 

10 may apply apparent forces to an input device where no linkage supports the input device 
relative to a fixed frame of reference, for example, when using exoskeletal gloves 
supported by the surgeon's arm. 

In general, non-visual information such as force which is sensed by the 
slave may be presented in corresponding non-visual formats (i.e., force reflecting 

15 master/slave arrangements), or in an alternative non- visual form (for example, force 
presented as sounds or heat). Non-visual information sensed by the slave may also be 
displayed to the surgeon in a visual format, such as using a bar graph or force vector, as 
described above. As used herein, non-visual information includes tactile sense 
information (including force, pressure, vibration, texture, heat, and the like), sound 

20 information (which may be sensed using a microphone of the slave), smell/taste (as may 
be sensed using a chemical or biochemical sensor of the slave), and the like. 

It should also be understood that traditionally graphical information 
(including optical images taken from an image capture device, ultrasound images, 
fluoroscopic images, tomographic images, and the like) may be presented both in a visual 

25 format (in the stereo viewer or other display mechanisms) and in non- visual formats (for 
example, using information sensed by ultrasound to identify, track, and avoid contact 
with selected tissues by imposing haptic walls in the simulated domain). Hence, non- 
visual information sensed by the slave and the non- visual information presented by the 
master may include a variety of information that is either sensed by and/or presented in a 

30 form for senses other than vision, including the sense of smell, the sense of taste, the 
sense of touch, the sense of hearing, and the like. 



WO 00/60521 

- 48 

As mentioned, the control system 810 enables limitations to be set in the 
simulation block 834. These limitations can be chosen to conform with mechanical 
system limitations or constraints and/or can be preset to correspond with 
environmentally-sensitive movement limitations at the surgical site as will be described in 
5 greater detail hereinbelow. Thus, the limitations imposed in the simulated domain 812, in 
one instance, can be regarded as virtual limitations corresponding with actual physical 
system limitations. The limitations at the simulated domain 812 are not derived from 
actual slave movement but from simulated or virtual slave movement. Thus, the slave is 
prevented from actually transgressing a limitation by simulating its movement and 

10 velocity and restricting the simulated movement and velocity before instructing the actual 
slave to respond. One typical limitation set in the simulated domain 812 concerns 
singularities of the system. 

What is meant by the term singularity will now be described by way of an 
example of a singularity in the mechanical structure of the minimally invasive surgical 

15 apparatus. Referring to Figure 2 A of the drawings, and as already mentioned, the 
instrument 14 when mounted on the robotic arm 10 is linearly displaceable in the 
direction of arrow P. If the instrument 14 is positioned such that the end effector 58 is 
relatively far removed from the fulcrum 49 and the master control is manipulated to 
command responsive movements, the responsive movement of the slave can normally 

20 readily be performed. At a specific fixed distance from the fulcrum 49, the end effector 
has a range of lateral movement constrained within bounds dictated by constraints in the 
mechanical structure of the arm 12. It will be appreciated that the closer the end effector 
58 is displaced toward the fulcrum 49, the smaller the possible range of lateral movement 
becomes. This can be visualized by picturing a cone having its apex at the fulcrum 49 

25 and extending from the fulcrum 49 in a downward direction in Figure 2A. The range of 
lateral movement of the end effector 58 being limited to within the visualized cone. It 
will thus be appreciated that toward the base of the visualized cone, e.g., a 1-inch lateral 
movement of the end effector, can normally readily be achieved by the mechanical 
structure of the arm 12. However, toward the apex of the cone, in other words toward the 

30 fulcrum 49, a point is reached where a 1-inch lateral movement of the end effector 58 is 
simply not achievable due to the mechanical constraints of arm 12. Furthermore, the 
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movement by the robotic ami 12 to induce lateral movement of the end effector 58 
becomes more radical the closer the end effector 58 is displaced toward the fulcrum 49. 

When a surgeon is performing a surgical procedure by means of the 
minimally invasive surgical apparatus, he or she is normally unaware of the robotic arm 

5 12 movements since he or she is viewing the surgical site through the viewer 202. 

Accordingly, unless provision is made to the contrary, it could happen that in the course 
of a surgical procedure the end effector 58 is displaced too near the fulcrum 49 so that 
master input causes the robotic arm 12 to move too quickly over corresponding long 
distances in responding to the commanded end effector movements. The control system 

10 81 0 is arranged to provide a method of inhibiting the robotic arm from making too rapid 
or large a movement in response to master input because of the singularity described 
above. 

Another singularity of the mechanical structure of the slave, in particular 
of the surgical instrument 14, will now be described with reference to Figure 5 of the 
15 drawings. 

As mentioned, the end effector 58 is angularly displaceable about axis 14.2 

* 

as indicated by arrows 59. Should the axis of symmetry 60A of the end effector be 
positioned along the axis 14.2, angular displacement of the end effector about axis 60A is 
readily induced. However, should the axis 60A be positioned perpendicular to the axis 
20 14.2, angular displacement of the end effector 58 about axis 60A is not possible. Thus, a 
singularity is approached as the axis 60A approaches a position perpendicular to the axis 
14.2. 

A further singularity of the robotic arm 10, can be understood with 
reference to Figure 4 of the drawings. As already mentioned, the robotic arm is angularly 
25 displaceable about axis 28 as indicated by arrows 26. When the axis 1 4.2 is 

perpendicular to the axis 28, movement of the arm 10 in the direction of arrows 26 is 
readily induced on the end effector 58. As will readily be observed in Figure 4, a 
singularity is approached the closer the axis 14.2 is moved toward a position parallel to 
the axis 28. 

30 Another typical limitation imposed in the simulated domain 8 1 2 relates to 

positional constraints of the various joints. 
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Another typical limitation imposed in the simulated domain is a velocity 
limitation corresponding to practicably mechanically achievable slave velocity. 
Naturally, the slave has greater mass and moments of inertia than the master. Thus, 
should the surgeon move the master too quickly, or should the master accidentally be 
5 knocked to induce rapid movement thereon, the slave would be commanded to move in 
sympathy with the master but at a rate not practicably achievable by the arm 10 due to 

mechanical constraints. 

Another limitation that may be imposed on movement of the master is for 
slaves and/or tools having limited degrees of freedom. While many tools will be 
1 0 provided with a full six degrees of freedom plus actuation (for grip, electrosurgical power 
on, etc.), other tools may have a more limited range of movement. For example, when an 
unarticulated tool (such as an endoscope or the like) is mounted to a manipulator so as to 
provide an end effector with five or fewer degrees of freedom, the processor may inhibit 
movement of the master to five or fewer corresponding degrees of freedom within the 
1 5 master controller workspace. As the master and slave will often be kinematically 

dissimilar, this may involve multiple coordinated master actuation torques to simulate a 
single "locked-out" degree of freedom of the slave. 

As mentioned, optionally, limitations relating to surgical environmental 
constraints can also be effected as described in greater detail hereinbelow. 
20 Referring now to Figure 1 6 of the drawings, one embodiment of the 

simulation block 834 includes a modified Jacobian inverse controller indicated by J" 1 * at 
870. The modified Jacobian inverse controller is arranged to inhibit the detrimental 
effects which result when a singularity is approached. This is achieved by modifying a 
Jacobian inverse matrix of the controller J"'*. The modification to the matrix will now be 
25 described by way of example and with reference to Figure 2A and 1 7 of the drawings. 
In Figures 2A and 17, the length of the arm portion of the shaft 14.1 of the instrument 14 
which extends beyond the fulcrum 49 is indicated by L. 

The relationship between velocity x in Cartesian space relative to angular 
velocity 9 in joint space is typically expressed by the relationship 
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For the minimally invasive surgical apparatus, the Jacobian matrix is 
typically in the form of a 6x6 term matrix for converting joint space coordinates to 
corresponding Cartesian coordinates. Naturally, some of the terms in the matrix include a 
multiplication factor equal to L. Accordingly, when it is desired to determine positions in 
joint space corresponding to Cartesian coordinates, the following relationship is used: 



0 =] A .x 



When the inverse Jacobian matrix is used in this fashion, the terms 
including the multiplication factor of L become terms having a multiplication factor of 

■ 

1/L. 

l 0 It will be appreciated that as L decreases the term 1/L approaches infinity. 

This characteristic associated with a singularity is schematically illustrated in Figure 17. 
The length L is indicated along the horizontally extending axis and the corresponding 
factor 1/L is indicated along the vertically extending axis. The parabolic lines indicate 
the relationship between L and 1/L. It is clear that when the desired joint velocity is 

1 5 determined by means of the Cartesian velocity x and a term includes the multiplication 
factor 1/L, the joint velocity approaches infinity as the value of L decreases, thus as the 
end effector is moved closer to the fulcrum 49. 

To compensate for these detrimental effects when a singularity is 
approached, the 1/L term in the Jacobian Inverse matrix is replaced with a function of L 

20 which yields a resultant relationship between L and 1/L as indicated in dashed lines in 
Figure 17. Two dashed lines are indicated to show different possible functions of L. In 
similar fashion the Jacobian Inverse matrix is modified to cater for all the singularities of 

the system already described. 

Referring again to Figure 16 of the drawings, the simulation block 834 will 

25 now be described in further detail. 

The modified Jacobian Inverse controller which makes allowance for 
singularities as hereinbefore described is indicated by the reference numeral 870. The 
Cartesian space reference velocity is input as indicated by arrow 833. After conversion to 
a resulting joint velocity by the controller 870, the resultant joint velocity is output at 874. 

30 The resultant joint velocity 874 is then input to a joint velocity limitation step at 876. At 
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this step the resultant joint velocity is limited to remain within a range between a 
predetermined maximum velocity V^, and a predetermined minimum velocity V™. 
These maximum and minimum values are typically selected to constrain the joint velocity 
within limitations corresponding to constraints of the mechanical structure of the system. 
5 Accordingly, at 876, should the joint velocity input 874 have a magnitude greater than the 
maximum and minimum values, the joint velocity magnitude 874 is decreased to within 
the set range. Thus: 

if 9 > max 6 = max 
if 0 < (min) b = (min) 

10 where 0 represents joint velocity, and max denotes a positive magnitude and min denotes 

a negative magnitude. 

After the joint velocity is limited in this manner, the joint velocity is 
integrated at 878 to yield a corresponding position in joint space. In similar fashion to the 
joint velocity limitation step at 876, the position is limited at 880 to remain within a set 

1 5 positional range. 

From 880, the resultant joint positional signal is routed to the filter 838 as 

indicated by one of the arrows 835 and as already described herein with reference to 

Figure 14. The resultant velocity signal as output from 876 is routed to the filter 838 as 

indicated by the other arrow 835. The resultant velocity signal is linked to the positional 

20 control step 880, as indicated at 881, so that in the event that the position is limited, the 

velocity signal is rendered zero. 

As mentioned, velocity, position and singularity limitations or constraints 
are applied to the Cartesian reference velocity in the simulation block 834 indicated in 
Figure 14 to yield a simulated slave joint position and velocity. Naturally, should the 
25 Cartesian reference velocity input to the simulation block 834 not result in a transgression 
of any of the limitations set for the slave, the Cartesian reference velocity input to the 
simulation block 834 is then simply transferred into corresponding slave joint position 
and velocity signals without any imposition of limitations. The corresponding slave joint 
position and velocity is then forwarded to the slave after the filtering step at 838. 
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An alternative simulation block 834B and another method of imposing 
limitations will now be described with reference to Figure 1 8 of the drawings in which 
like reference numerals are used to designate similar parts unless otherwise indicated. 

Referring now to Figure 18, and in the simulation block 834B, the 
5 Cartesian reference velocity is initially input into a Cartesian position and velocity limit 
block at 902. At 902, any desired limitations to position and velocity in Cartesian space 
can be set. This can be achieved in similar fashion to the manner in which the joint 
velocity and position limitations were imposed in Figure 16. Such limitations can be 
chosen to suit the specific surgical procedure to be performed. Thus, for example, should 
10 the surgical procedure to be performed be at a sensitive location, such as close to the 
brain, or heart, or the like, limitations can be set to constrain end effector movement 
within a space so as not to be able to contact the area of sensitivity. Thus, at 902, 
limitations can be tailored to meet specific environmental limitations defined by the 
specific surgical procedure to be performed so as to avoid accidental damage to a 
15 sensitive organ, or the like. Thus, at 902, slave position and velocity can be restricted to 
remain within preset limitations dictated by the surgical procedure to be performed. It 
will be appreciated that such surgical environment dependent limitations can be imposed 
in the simulation block 834 in Figure 16, and also in the preferred simulation block 834A 
to be discussed with reference to Figure 20. 
20 After the limitation step at 902, the resultant Cartesian velocity is input to 

a modified Jacobian Inverse controller at 904. The modified controller 904 imposes 
limitations on the Cartesian velocity input during conversion of the Cartesian velocity 
input into a corresponding joint space velocity to make allowance for singularities as 
already described. 

25 From the modified Jacobian Inverse controller 904, the resultant joint 

velocity is input into a joint position and velocity block at 906. At the joint position and 
velocity block 906, the joint velocity input is monitored to ensure that corresponding 
velocity and position commands to each specific joint would not transgress set limitations 
corresponding to actual angular position and velocity limitations of that joint. After the 

30 joint velocity has been monitored at 906, and any limitations imposed, the resultant 
simulated slave joint velocity is output as indicated by arrow 835. The simulated slave 
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joint velocity is also fed through an integration step at 91 0 to yield the corresponding 

simulated slave joint position. 

The simulated joint position for each specific joint is routed to the joint 

position and velocity block 906, and the modified Jacobian Inverse block 904 as indicated 
in dashed lines. The position signal 835 is routed to the modified Jacobian Inverse block 
904 to enable transformation from Cartesian to joint space. The position signal 835 is 
routed to the position and velocity block 906 in order that joint position and velocity 
limits can be imposed at 906. This will now be described with reference to Figure 1 9 in 
which like reference numerals are used to designate similar parts unless otherwise 
indicated. It will be appreciated that Figure 19 exemplifies the imposition of positional 
and velocity limits on a single joint. The same method of imposing such positional and 
velocity limits is employed for each joint at 906. 

In Figure 19, the joint velocity input from the modified Jacobian Inverse 
controller at 904 is indicated by arrow 912. The resultant velocity after having passed 
15 through the joint position and velocity block is indicated by arrow 914 and the joint 

position input is indicated by arrow 835 and is shown in dashed lines. The joint for which 
position and velocity limits are to be imposed by the block diagram shown in Figure 19 
normally has physical limitations. Thus, the joint has a maximum position in which the 
arm members which are pivotally connected thereby are at a maximum angular position 
20 relative to each other. Similarly, the joint has a minimum position in which the arm 

members which are connected one to another thereby are at a minimum angular position 
relative to each other. Accordingly, the joint has an angular displacement range 
extending between its minimum and its maximum position. The angular limits of the 
joint are indicated by blocks 918 and 920, respectively, block 918 indicating the 
25 minimum position and block 920 the maximum position. Naturally, since we are dealing 
with a simulated domain, the limits can be chosen to suit. Accordingly, the minimum and 
maximum angular positions 91 8, 920 need not necessarily correspond with the actual 
physical positional limits of the joint, but can be chosen at any appropriate angular 
positions within the angular positional range capabilities of the joint. 
30 The position input at 835 is normally varying continually as the surgeon 

manipulates the master during the course of a surgical procedure. The positional input 
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835 is fed to to sunken junctions 922, 924. A, to junction 922, the angular position 
as i„p„, a. 835 is conned with the positional minimum or lower limit to yteld an 
an Jar valno corresponding ,0 the angular deviation of tine position input 835 
JL, 9. 8. flrus, at 922, an angtdar va!ue e,ual to tite difference between the angular 
5 limi ,anddreangu,arpo Si tioninpu,835isdete mi ned. 

l0W er timi. 918 thus detemtined, is then fed to a velocity delation b.ock M » * 
processing cycle rate of the control system is known, fa tins case, ,. ts W ,cm.y .300 H, 
A. 926 the velocity which the join, needs to have to cause its position to ctno.de wtth 
to lower join, tinti, 9,8 a, the nex, processing cycle is detennined. This ve,oei„ value ts 
,0 then routed to a decision Woe* a, 928. Nattnatiy, if tite angmar position as mpu a, 835* 
far removed from tite lower timi, 91 8, the resultan, ve.oci,y van., denved a, 926 wall be 
very .arge, and typically physical* unattainable. However, as tite 
approaches zero, namely, where the anguUr position 835 approaches to .ower 1 tmt *8, 
2 velocity output from 926 becomes !ess than me finable join, veloctty a^ becomea 
,5 zerowheretoangularposition835isa.tolowerlimi.918. 

Reference numeral 930 represents a set joint velocity limit Thts hmt, ts 
.ypicatiy chosen in accordance with me accept join, velocity W of that join,. Thts 
se, velocity lower hmi, is also fed into the decision block 928. A, 928 me two mm 
verities are compared and me .argea, of me W o selected. It wil. be apprecated to, the 
20 Urges, value is ae.ec.ed because we are regarding a velocny timi, in a negative dtreetion. 
Thus me .arges, value is me same as me smallest absolute va.ue. The selected veloctty 
value Otus determined defines me lower velocity limit as indicated a, 932. 

It could happen that the joint is positioned beyond the positional lower 
tai.918 This can occur when me minimally invasive surgical apparatus is inmally 
25 setup, or where me positiona. limits are selectively changer!, for example. In such a case, 
it ,, deshable to cause me join, position to relum to within me range se, by me upper and 
.ower timi* a. 918 and 920, respectively. For the .owes angular position to.., tins . 
achieved by .he Mock 934. What is achieved by the block 934, is a constant curbmg of 
positional movement beyond the lower limit. Thus, as tite surgeon manipulates to 
30 master, movements causing to angular position of the join, to move toward to tan,, are 
permitted, but once such movement has taken place, to join, is restricted to tts new 
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position closer to the limit. The process is maintained until the joint position is W ithin the 
range set by the values at 918, 920, respectively. 

It will be appreciated that a maximum velocity, as indicated by reference 
numeral 935 is determined in similar fashion as the minimum velocity, as can be seen in 

5 Figure 19 of the drawings. 

Referring now to Figure 20 of the drawings, a preferred simulation block 
834A will now be described. In Figure 20 the same reference numerals are used to 
designate similar parts or aspects unless otherwise stated. 

In Figure 20, the Cartesian reference velocity is input as indicated by 
10 arrow 833. The simulated joint positions and velocities are output at 835. The Cartesian 
reference velocity 833 is routed to a modified full Jacobian Inverse block at 942 and to an 

isolation block at 944. 

At 942, the Cartesian reference velocity signal 833 is transformed into a 

corresponding joint velocity signal 946. The modified full Jacobian Inverse block 942 
15 makes allowance for singularities as already described with reference to 904 in Figure 18. 

In the minimally invasive surgical apparatus under discussion, the 
modified full Jacobian Inverse block typically includes a six by six term matrix. After 
transformation at the block 942, the resultant joint velocity signal is passed to an isolation 
block 948 At the isolation block 948, the terms relating to the wrist joints, as indicated m 
20 Figure 5 of the drawings, are isolated from the terms relating to the joints on the robotic 
arm 12, as indicated in Figure 2A and 2B. After isolation at 948, the wrist joint velocities 
are forwarded to a wrist joint velocity and position limitation block at 950. 

At 950 wrist joint velocity limits are imposed on each wrist joint in similar 
fashion to the method described above with reference to Figure 19. However, for the 
25 wrist joints, namely the joints providing the three degree of freedom of movements to the 
end effector 58, the limitations are imposed simultaneously rather than on a joint by joint 
basis This will now be described with reference to Figure 21 . 

Referring to Figure 21, the limits for each joint are determined in similar 
fashion to that described with reference to Figure 19. But, as indicated at 970, the 
30 limitations are used to define a corresponding velocity limitation for the three joints 
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together as indicated by the box 972. Accordingly, a multidimensional join, velociry 
limitation, in this case a thme^imensional joint velocity limitation, isprovtded. 

The input joint velocity signal at 951 is compared to the multidimenstonal 
join, velocity .imitation, a, 970. Should the input velocity signal 951 fall entirely instde 
the limitation, it is unchanged by me limitation. In such a case the output velocity atgnal 
952 is me same as the input velocity signal 95 1 . However, should the input veloct.y 
signal 951 611 outside me limitation, me limitation block a. 970 will select the output 
velocity 952 according to a criterion, which will now be described. 

A joint velocity error between the input velocity signal 951 and the 
se.ec.ed output velocity 952 is defined as iHustrated a. 974. The join, velocity error is 
transformed into a Cartesian vdoci.y error using a Jacobian matrix a. 976. It wtll be 
appreciated mat me Jacobian matrix a, 976 describes me kinematics of the wristjomts, 
which includes pivots 54, 60 and axis 14.2, with reference to Figure 5. The magnitude of 
the Cartesian velociry error is then determined at 978. 

The criterion for selection of the output velocity 952 by the limitation 
block 970 is me obedience of the multidimensional limitation and me minimization of me 

Cartesian velocity etror magnitude. 

Returning now to Figure 20 the drawings, the output 952 from the 
.imitation block 950 represents a combined join, velociry signal including join, velocities 
a, me join* or pivo.s 54, 60 and join, velocity abon. axis 14.2, with reference to Figure 5 
of ft. drawings, after any limitations relating .0 velocity, position and singularities have 
been imposed. 

At the isolation block 944, the translation* Cartesian velocity terms are 



25 



30 



At the isolation uiui^n. ttt, « 

isolated from me Cartesian reference velociry signal 833. The isolaied ,erms correspond 
,„ the Cartesian velocity commands addressing tire joints on me robotic arm 12. After 
isolation, me Cartesian reference velociry signal for me ou.er join* only is forwarded ,o 

an adjustment block at 954. 

In the event that the wrist joint velocity signal was restricted at one or both 
O ftheblocks942,950,theouterjointvelocitycanbeadaptedat954. This will now be 

described in greater detail and with reference to Figure 5 of the drawings. 
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It will be appreciated that a command at the master control 700 relating to 
only an orientation change of the end effector 58 can result in not only responsive angular 
m0 vement about pivots 54, 60 and about axis 14.2 but also responsive outer joint 
m0 vement. This is so because of structural dissimilarities between master and slave. 
Thus for the slave to perform an orientational movement corresponding to a master 
optional movement, it is sometimes desired for the slave outer joints to move aha, 

Accordingly, in the event that wrist joint velocity limits were imposed, it is 
desired to adapt outer joint, or translation*, velocity to the extent to which the outer joint 
velocity formed part of the orientational wrist limitation. This is achieved at 954. 

The resultant, possibly adapted, translation^ Cartesian velocity s lg nal is 
th en forwarded to a modified translation Jacobian Inverse block at 956. At 956 the 
signal is converted into a corresponding joint space velocity signal. The modified^ 
Jacobian Inverse matrix at 956 makes allowance for the fulcrum 49 singularity and the 
ma ximum robotic arm pitch singularity as already described with ~ » ^ 
The joint space velocity signal from 956 is then passed to a limitation block at 958 At 
958 positional and velocity limitations are imposed on the signal in a manner similar to 
that already described with reference to Figure 19 of the drawings, and for each outer 

JOmt ' The final wrist joint velocity signal and the final outer joint velocity signal 

20 arethencombinedat960toyieldthesimulatedjointvelocity835. The simulated joint 
velocity 835 is integrated at 962 to yield a corresponding simulated joint position, 

indicated by the other of the arrows 835. 

The simulated joint position is fed to the blocks 942, 950, 954, 956 and 

95 8 to enable the desired computations. 

While this invention has been particularly shown and described with 
references to preferred embodiments thereof, it will be understood by those skilled in the 

« fx™ and details may be made thereto without departing from 
art that various changes in form and details may uc 

the spirit and scope of the invention as defined in the accompanying claims. For 
example, while the invention describes the use of Cartesian coordinate systems, the 
30 invention may also find applications taking advantage of polar coordinate systems, 
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cylindrical coordinate systems, or the like. Hence, to scope of .he presen. invention is 
limited solely by the following claims. 



WO 00/60521 



- 60 - 



PCT/USOO/08526 



^HATTSPT AIMED IS: 

\ A surgical robotic system comprising: 
2 a master controller having an input device movable in a controller 



3 workspace; 
4 



8 



a slave having a surgical end effector and at least one actuator operatively 

5 eoupled to the end effector, the actuator moving the end effector in a surgical workspace 

6 in response to slave actuator signals; 

7 an imaging system including an image capture device with a field of view 
movable in the surgical workspace, the imaging system generating state variable signals 

9 indicating the field of view; and 

10 a processor coupling the master controller to the slave arm, the processor 

1 1 generating the slave actuator signals by mapping the input device in the controller 

12 workspace with the end effector in the surgical workspace according to a transformation, 

13 the processor deriving the transformation in response to the state variable signals of the 

14 imaging system. 

! 2. The surgical robotic system of claim 1 , wherein the image capture 

2 device moves independently of the slave. 



1 

2 
3 



3. The surgical robotic system of claim 2, wherein the processor 
derives the transformation so that an image of the end effector in the display appears 
substantially connected to the input device in the controller workspace. 

j 4. The surgical robotic system of claim 1 , wherein the slave generates 

2 state variable signals responsive to a position of the end effector in the surgical 

workspace, the processor deriving the transformation using the state variable signals of 

4 the slave. 

5. The surgical robotic system of claim 4, wherein the processor 
determines aposition and orientation of the input device in the master controller space 



3 



1 



2 



3 from state variable signals of the master controller, wherein the processor determmes a 

4 position and orientation of the end effector in the surgical workspace from the state 
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5 variable signals of the slave, and wherein the processor generates the slave actuator 

6 signals by comparing the position and orientation of the input device and the end effector 

7 in the mapped space. 

1 6. The surgical robotic system of claim 4, wherein the slave 

2 comprises an elongate shaft supporting the end effector, wherein the image capture device 

3 comprises an endoscope having a distal end, and wherein the processor derives the 

4 transformation when the shaft and endoscope are inserted into the surgical site through 

5 arbitrary minimally invasive access sites so that the end effector and the distal end of the 

6 endoscope move by pivoting the shaft and the endoscope about the access sites. 

1 7. The surgical robotic system of claim 1 , wherein the imaging 

2 system comprises a linkage movably supporting the image capture device, the slave 

3 comprising a linkage movably supporting the end effector, the linkages comprising joints 

4 and the state variable signals comprising joint configuration signals, the linkages coupled 

5 so that the processor derives the transformation in response to the joint configuration 

6 signals such that movement of an image of the end effector in the display appears 

7 substantially connected to the input device in the workspace. 



1 8. The surgical robotic system of claim 7, wherein the slave linkage is 

2 mounted to a slave base, and wherein the imaging system linkage is mechanically coupled 

3 to the slave base. 

1 9. The surgical robotic system of claim 7, wherein the imaging 

2 system linkage is mounted to the slave base, and wherein the slave base has wheels for 

3 transporting the end effector linkage and the image capture device. 

1 1 0. The surgical robotic system of claim 1 , wherein the slave 

2 comprises a tool holder and a tool detachably mounted in the tool holder, the tool 

3 including the end effector and at least one joint, the surgical robotic system further 

4 comprising a plurality of alternative kinematically dissimilar surgical tools having 

5 alternative surgical end effectors, the alternative tools mountable to the at least one tool 



WO 00/60521 PCT/US00/08526 



- 62 - 

6 holder in place of the tool, the processor capable of changing the transformation in 

7 response to a tool change signal. 

1 11. The surgical robotic system of claim 1 , further comprising a 

2 plurality of slaves, the master controller selectively associatable with the slaves in 

3 response to a slave selection signal, wherein the processor changes the transformation in 

4 response to the slave selection signal so that movement of an image of the end effector of 

5 the selected slave as shown in the display substantially corresponds to movement of the 

6 input device in the workspace. 

1 1 2. The surgical robotic system of claim 1 , wherein the processor 

2 derives the transformation in real time. 

1 13. A surgical robotic system comprising: 

2 a master controller having an input device movable in a controller 

3 workspace; 

4 a slave having a surgical end effector and at least one actuator operatively 

5 coupled to the end effector, the actuator moving the end effector in a surgical workspace 

6 in response to slave actuator signals; 

7 an imaging system including an image capture device with a field of view 

8 movable in the surgical workspace, the imaging system transmitting an image to a 

9 display; and 

I o a processor coupling the master controller to the slave arm, the processor 

I I generating the slave actuator signals by mapping the input device in the controller 

12 workspace with the end effector in the surgical workspace according to a transformation, 

13 the processor deriving the transformation so that an image of the end effector in the 

14 display appears substantially connected to the input device in the workspace. 

1 14. The surgical robotic system of claim 13, wherein the master 

2 controller comprises a linkage supporting the input device and the slave comprises a 

3 linkage supporting the end effector, the master linkage and the slave linkage being 

4 kinematically dissimilar. 
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15. The surgical robotic system of claim 1 4, wherein joints of the 



2 master linkage and joints of the slave linkage have different degrees of freedom. 

1 1 6. The surgical robotic system of claim 1 3 , wherein joints of the 

2 master linkage and joints of the slave linkage define different locations in the mapped 

3 space. 

1 17. The surgical robotic system of claim 13, wherein: 

2 the slave comprises a linkage with a plurality of joints supporting the end 

3 effector, the actuators coupled to the joints so as to pivot a rigid shaft of the slave about a 

4 first arbitrary minimally invasive access and move the end effector in response to slave 

5 motor signals; and 

6 the imaging system comprises actuators and a linkage supporting the 

7 image capture device having, the linkage comprising a plurality of joints, the actuators 

8 coupled to the joints to move the field of view within the surgical workspace when a rigid 

9 shaft of the image capture device extends through a second arbitrary minimally invasive 
10 access. 

1 1 8 . The surgical robotic system of claim 1 3 , wherein at least one of the 

2 slave, the master controller, and the imaging system comprise a linkage having a 

3 detachable connection disposed between joints so as to define a first linkage portion 

4 attached to a base and a second linkage portion attached to the first linkage portion, the 

5 linkage portions defining first and second coordinate reference systems, wherein the 

6 processor derives the transformation indirectly from a first relationship of the first 

7 coordinate system to the base of the first linkage system and from a second relationship of 

8 the second coordinate system to the first coordinate system. 

1 19. The surgical robotic system of claim 13, wherein the processor 

2 calculates the transformation in response to a signal indicating at least one member of the 

3 group consisting of a movement of the camera, a decoupling and repositioning of one of 

4 the master and the slave relative to the other, a tool change mounting a different end 
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effector on the slave, a change in scale of the mapping, manual movement of a passive 



6 j oint of the master or slave, and association of the master with an alternative slave. 



20. The surgical robotic system of claim 1 3, wherein the slave senses 

2 non-visual sensory information at the surgical workspace, the master controller presenting 

3 the non-visual information to a surgeon manipulating the input device. 



21. The surgical robotic system of claim 20, wherein the display shows 



2 a graphical representation of the non-visual information. 



22 . The surgical robotic system of claim 2 1 , wherein the non-visual 



2 information comprises force applied at the end effector. 



23. The surgical robotic system of claim 22, wherein the display 

2 represents the force as a member selected from the group consisting of a bar graph, a 

3 force vector, and an end effector color. 



24. The surgical robotic system of claim 20, wherein the non-visual 



2 information comprises force applied at the end effector, and wherein the master controller 

3 comprises a sound generator, the sound generator producing a sound varying in response 



4 to the force. 



25. The surgical robotic system of claim 13, wherein the master 

2 controller presents non-visual sensory information to a surgeon manipulating the input 

3 device. 

1 26. The surgical robotic system of claim 25 , wherein the master 

2 controller presents the non-visual information with an orientation correlating to the 

3 image. 

1 27. The surgical robotic system of claim 25, wherein the master 

2 controller comprises a plurality of actuators, the actuators providing tactile feedback to 

3 the surgeon manipulating the input device in response to master actuator signals 

4 generated by the processor. 
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1 28. The surgical robotic system of claim 27, wherein the actuators 

2 apply loads against the input device in response to the master actuator signals, and 

3 wherein the processor generates the master actuator signals in response to a comparison 

4 between a position and orientation of the end effector and a position and orientation of the 

5 input device in the mapped space. 



1 29. The surgical robotic system of claim 26, wherein the non-visual 

2 information indicates forces and torques applied to the slave. 

1 30. The surgical robotic system of claim 29, wherein the non-visual 

2 information comprises forces and torques applied via the input device to the hand of the 

3 surgeon so that the input device forces and torques substantially correspond to the slave 

4 forces and torques according to the image of the slave shown in the display. 

1 31. The surgical robotic system of claim 29, wherein a correlation 

2 between the non-visual information and the image is revised by the controller when the 

3 transformation is revised. 

1 32. The surgical robotic system of claim 1 3, wherein the processor 

2 generates the slave actuation signals so that at least one angle selected from the group 

3 comprising a change in angular orientation and an angle of translation of the end effector 

4 remains within 5 degrees of the at least one angle of the input device. 

1 33. A surgical robotic system as claimed in claim 32, wherein an input 



2 device movement defines an input movement distance, and wherein the image of the end 

3 effector moves an output movement distance in response to the input movement distance, 

4 the output movement distance being significantly different than the input movement 

5 distance. 

1 34. A surgical robotic system comprising: 

2 a master controller having an input device supported by a linkage so that 

3 the input device can move in a controller workspace with a first number of degrees of 

4 freedom; 
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5 a slave having a surgical end effector and a plurality of actuators 

6 operatively coupled thereto so that the end effector can move in a surgical workspace 

7 with a second number of degrees of freedom in response to slave actuator signals, the 

8 second number being less than the first number; and 

9 a processor coupling the master controller to the slave, the processor 

1 0 generating the slave actuator signals by mapping the input device in the controller 

1 1 workspace with the end effector in the surgical workspace. 

1 35. The surgical robotic system of claim 34, wherein the master has at 

2 least one redundant degree of freedom. 

1 36. The surgical robotic system of claim 34, wherein the slave 



2 comprises a manipulator arm releasably supporting the end effector and further 

3 comprising an alternative end effector mountable to the arm in place of the end effector, 

4 the alternative end effector having at least one more degree of freedom than the end 

5 effector, wherein the processor inhibits movement of the input device in the controller 

6 workspace when the end effector is in use so that the input device is movable in the 



7 second number of degrees of freedom. 

1 37. A surgical robotic system comprising: 

2 a master controller having an input device movable in a controller 

3 workspace; 

4 a slave comprising a slave arm and a first tool releasably mountable on the 

5 arm, the first tool having a first end effector movable in a surgical workspace in response 

6 to slave actuator signals; 

7 a second tool releasably mountable on the slave in place of the first tool, 

8 the second tool having a second end effector movable in the surgical workspace in 

« 

9 response to the slave actuator signals, the second tool being kinematically dissimilar to 

10 the first tool; and 

1 1 a processor coupling the master controller to the slave arm, the processor 

12 generating the slave actuator signals by mapping the input device in the controller 

13 workspace with the end effector of the mounted tool in the surgical workspace. 
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1 38. A surgical robotic system comprising: 

2 a master controller having an input device movable in a master controller 

3 space, the input device having a grip sensor for squeezing with a hand of a surgeon, the 

4 grip sensor defining a grip pivot; 

5 a slave arm having an end effector supported by a linkage so that the end 

6 effector is movable in a surgical workspace, the slave arm having actuators coupled to the 

7 linkage for moving the end effector in response to slave actuator signals, the end effector 

8 comprising jaws with a jaw pivot; 

9 an image capture device having a field of view within the surgical 

10 workspace and transmitting an image to a display; and 

1 1 a processor coupling the master controller to the slave arm, the processor 

12 generating the slave actuator signals in response to movement of the input device so that 

13 the jaw pivot in the display appears substantially connected with the grip pivot. 

1 39. The robotic system of claim 38, wherein the end effector rotates 

2 about the jaw pivot when the input device rotates about the grip pivot so that an 

3 orientation of the end effector image shown in the display substantially corresponds to an 

4 orientation of the input device in the controller workspace. 

1 40. A surgical robotic method comprising: 

2 moving a master input device in a controller workspace by articulating a 

3 plurality of master joints; 

4 moving a surgical end effector in a surgical workspace by articulating a 

5 plurality of slave joints in response to slave motor signals; 

6 displaying , on a display, an image of an arbitrary field of view within the 

7 surgical workspace adjacent the master controller; and 

8 automatically generating the slave motor signals in response to moving the 

9 master so that an image of the end effector shown in the display appears substantially 
10 connected with the input device in the master controller space. 

1 4 1 . A surgical robotic method comprising: 



WO 00/60521 PCT/US00/08526 

- 68 - 

2 moving a master input device in a controller workspace by articulating a 

3 plurality of master joints; 

4 generating master joint signals responsive to the master joint 

5 configuration; 

6 moving an end effector in an end effector workspace by articulating a 

7 plurality of slave joints in response to slave motor signals; 

8 generating slave joint signals responsive to the slave joint configuration; 

9 calculating a position of the input device in the master controller space 

1 0 from the master joint signals; 

1 1 calculating the end effector position in the end effector workspace from 

12 the slave j oint signals; 

1 3 mapping the end effector workspace with the controller workspace 

14 according to a transformation; 

1 5 generating the slave motor signals in response to a difference between the 

16 input device position and the end effector position in the mapped space; 

1 7 moving a field of view of an image capture device in the end effector 

1 8 workspace by articulating a plurality of image device joints; 

19 displaying an image of the field of view adjacent the master controller and 

20 generating image device signals responsive to the image device joint configuration; and 

21 revising the mapping in response to the image device signals so that 

22 movement of an image of the end effector shown in the display remains aligned with the 

23 movement of the input device in the master controller space. 

1 42. A surgical robotic system comprising: 

2 a master controller having an input device movable in a master controller 

3 space, the input device including first and second grip members for actuating with first 

4 and second digits of a hand of an operator; 

5 a slave having a surgical end effector, the end effector moving in a surgical 

6 workspace in response to slave actuator signals and including first and second end 

7 effector elements; and 
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8 a processor coupling the master to the slave, the processor generating the 

9 slave actuator signals so that movement of the first and second grip members 
10 substantially map movement of the first and second end effector elements. 

1 43. The surgical robotic system of claim 42, wherein a pivotal joint 

2 between the first and second grip members is substantially connected to a pivotal joint 

3 between the end effector elements. 

1 44. The surgical robotic system of claim 42, wherein a midpoint 

2 disposed between tips of the first and second grip members is substantially connected to a 

* 

3 midpoint between tips of the end effector elements. 

1 45. A surgical robotic system comprising: 

2 a master controller having a handle supported by a plurality of joints so 

3 that the handle is movable in a master controller space, the joints defining a gimbal point 

4 of rotation about a plurality of axes, the handle disposed adjacent the gimbal point; 

5 a slave having a surgical end effector, the end effector moving in a surgical 

6 workspace in response to movement of the handle. 

1 46. The surgical robotic system of claim 45, wherein the end effector is 



2 supported by a plurality of joints, a last joint being disposed adjacent the end effector, and 

3 further comprising a processor coupling the master to the slave, the processor generating 

4 slave actuator signals so that the gimbal point of the master is substantially connected to 

5 the last joint of the slave. 



1 47. A surgical robotic system comprising: 

2 a master controller having a handle, the handle moving in a master 

3 controller workspace; 

4 a slave supporting a surgical end effector, the slave moving the end 

5 effector within a surgical workspace in response to slave actuation signals; and 

6 a processor coupling the master to the slave, the processor generating the 

7 slave actuation signals so that movement of a mapping point along the handle of the 

8 master controller substantially maps movement of a mapping point along the end effector, 
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9 the processor capable of changing at least one of the handle mapping point and the end 
1 0 effector mapping point. 

1 48. The surgical robotic system of claim 47, wherein the end effector 

2 includes a first end effector releasably mounted to the slave and a second end effector 

3 releasably mountable to the slave in place of the first end effector, the first end effector 

4 having a first length from a first end effector pivot, the second end effector having a 

5 second length, and wherein the processor can change the at least one mapping point to 

6 adjust for a difference between the first and second lengths. 

■ 

1 49. The surgical robotic system of claim 47, wherein the handle 

2 comprises first and second grip members squeezable about a pivot, wherein a grip axis is 

3 disposed on a midline between the first and second grip members and the handle mapping 

4 point remains disposed between the grip pivot and an end of the handle on the handle axis 

5 wherein the end effector comprises jaws with an end effector axis disposed on a midline 

6 between first and second jaw elements, and wherein the end effector mapping point 

7 remains disposed between the end effector pivot and an end of the jaw elements along the 

8 end effector axis. 



1 50. The surgical robotic system of claim 47, further comprising a 

2 display disposed adjacent the master, the display showing an image of the end effector 

3 with a magnification, wherein the at least one mapping point moves with changes in the 

4 magnification of the display. 

1 5 1 . A surgical robotic method comprising: 

2 moving a master input device in a controller workspace by articulating a 

3 plurality of master joints; 

4 generating master joint signals responsive to the master joint 

5 configuration; 

6 moving an end effector in an end effector workspace by articulating a 

7 plurality of slave joints in response to slave motor signals; 

8 generating slave joint signals responsive to the slave joint configuration; 
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9 calculating a position of the input device in the master controller space 

1 0 from the master joint signals; 

1 1 calculating the end effector position in the end effector workspace from 

12 the slave joint signals; 

1 3 mapping the end effector workspace with the controller workspace 

14 according to a transformation; and 

15 generating the slave motor signals in response to a difference between the 

16 input device position and the end effector position in the mapped space. 
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